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ABSTRACT 



This report describes and evaluates a training 
program for educational researchers conducted prior to and following 
the 1969 annual meeting of the American Educational Research 
Association. The report’s description of each of the program’s 12 
specific training sessions, which served a total of 542 educational 
researchers, includes the following elements: (1) Session title and 

staff, (2) general description, (3) objectives, (4) schedule, (5) 
participant characteristics, (6) instructional and evaluation 
materials, and (7) evaluation by staff and participants. Training 
sessions described cover the relationship of the following topics to 
educational research: (1) Instructional product (materials) 

development, (2) nonparametric methods and asscjiated post hoc 
procedures, (3) the computer and natural language, (4) methods for 
improving children’s learning proficiency, (5) systems approach in 
counseling and counselor education, (6) multivariate design and 
analysis, (7) anthropological methods, (8) sample free test 
calibration and person measurement, (9) survey research, (10) 
multiple group discriminant strategy, (11) Baysian statistical 
analysis, and (12) design and analysis of comparative experiments. 
(JH) 
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INTRODUCTION 



During February 1 to 5, 1969 the American Educational Research Asso- 
ciation (AERA) conducted a program of eight research training preses- 
sions prior to the annual meeting of the Association in Los Angeles. 
Approximately two months later, March 28 to April 1, four research train- 
ing postsessions were staged by AERA in College Park, Maryland. The com- 
bined presessions and postsessions served 5^2 educational researchers. 

The costs of the program were borne by AERA, the U.S. Office of Educa- 
tion, and the participants themselves. This report describes the back- 
ground, selection, planning, conduct, and evaluation of the 1969 AERA 
Research Training Sessions. 



BACKGROUND OF THE RESEARCH TRAINING SESSIONS 

The 1969 presession and postsession programs can trace their ori- 
gins to informal meetings of one or two days duration involving a rel- 
ative handful of selected researchers prior to the 1964- and 1965 annual 
AERA meetings. The 1964 and 1965 informal meetings were not widely 
publicized and really did not have the training of researchers as their 
primary mission. However, they can be regarded as the precursors of 
the AERA. research training sessions since in 1966 the prototypical "pre- 
session" was held as one of a group of three meetings in the tradition 
of these previous preconvention meetings. The 1966 pre session which set 
a pattern, thereafter adopted for the AERA presession programs, was a 
session dealing with experimental design under the direction of Richard 
E. Schutz. This 1966 presession on experimental design was the first 
five-day presession sponsored by AERA and was, in addition, the first 
formal research training program completed under Title IV of the Elemen- 
tary and Secondary Education Act of 1965 • Further, the session directed 
by Schutz was the initial presession in connection with a professional 
meeting that was systematically evaluated^- with respect to the attain- 
ment of its objectives. 

Because of the success and acceptance of the 1966 prese 3 Sion on 
exper ime ntal design, coupled with a growing interest of AERA m em bers 
in the possibility of expanding and formalizing other presession meet- 
ings, AERA sponsored a program of six courses in the 1967 presession 
program under the general chairmanship of Richard E. Schutz. These ses- 
sions and their directors were the following: 

1. Bayesian Statistical Analysis 

Donald Meyer, Syracuse University 

2. Curriculum Research and Evaluation 

Robert L. Baker, Arizona State University 

W. James Popham, University of California, Los Angeles 



^■Stake, R. E.$ Glass, G. V; and Taylor, P. A. Report of the Eval- 
uation of the AERA 1966 Presession on the Design of Educational Exper- 
iments. Center for Instructional Research and Curric ul u m Evaluation, 
University of Illinois, 1966. 95 pages (mimeograph). 



3 » Design and Analysis of Comparative Experiments in Education 
Gene V Glass, University of Illinois. 

4 . Educational Research Management Procedures 
Desmond Cook, Ohio State University 

Multivariate Design and Analysis in Educational Research 
Joe Ward, Southwest Educational Development Laboratory 

6 . Research Strategies with Culturally Deprived Children 
Martin Deutsch, New York University 

The 1967 pre session program was supported in part by a grant from 
the U.S. Office of Education under Title IV of ESEA, 1965* Approxi- 
mately 500 researchers applied for the program and somewhat more than 
300 researchers actually participated. 

The response to the extended 1967 program was highly positive and 
led to a much expanded program of eleven presessions in 1968 under the 
chairmanship of Gene V Glass. The 1968 sessions were the following: 

1 . Research in Reading Instruction 

John R. Bormuth, University of Chicago 

2 . Educational Research Management Procedures 

Desmond L. Cook, Ohio State University 

3. Anthropological Field Methodology in the Study of Education: 
With Particular Emphasis on Classroom Behavior and School 
Admini str at ion 

Frank W. Lutz, New York University 

4 . Nonparametric Methods in Educational Research 

Leonard A. Marascuilo, University of California, Berkeley 

5. Design and Analysis of Comparative Exper im ents 

Jason Millman, Cornell University 

6 . Evaluation: New Concepts in Scope, Strategy and Purposes 

C. Robert Pace, University of California, Los Angelas 

7 « The Computer and Natural Language 

Ellis B. Page, University of Connecticut 

8 . Instructional Product Research 

W. James Popham, University of California, Los Angeles 
Howard Sullivan, Southwest Regional Laboratory for Educa- 
tional Research and Development 

9 « On-line Computer Applications in Educational Research 

Ronald G. Ragsdale, Ontario Institute for Studies in 

Education 



10. Multivariate Design and Analysis in Educational Research 

Joe H. Ward, Southwest Educational Development Laboratory 

11. Developmental Processes in College Students 

Jonathan R. Warren, Educational Testing Service 

The 1968 presessions program was also supported in part by a grant 
from the U.S. Office of Education under Title IV of ESEA, 1965. Nearly 
750 individuals applied for the program and approximately 550 actually 
participated. 



THE 1969 RESEARCH TRAINING SESSI ONS 

In the late spring of 1969 AERA president David R. Krathwohl ap- 
pointed the following Research Training Presessions Committee: 

W. James Popham, Chairman 

Samuel Goldman, Division A - Administration 

Roland Payette, Division B - Curriculum and Objectives 

Leonard Maras cuilo. Division C - Learning and Instruction 

S. David Farr, Division D - Measurement and Research Methodology 

Carl Thoresen, Division E - Student Development and Personnel 

Services 

Charles Burgess, Division F - History and Historiography 

The Call for Proposals . In April, 1968 the call for 1969 preses- 
sion proposals was distributed to the membership primarily through an 
announcement that appeared on the front page of the Educational Re- 
searcher, the official newsletter of the American Educational Research 
Association (Number 5, 1968). The text of that call for presession 
proposals is presented below: 

Potential directors for the 1969 AERA Pre sessions, to 
be held on February 1 - 5> 1969> are being sought among the 
ranks of AERA members. The Presession Committee plans to 
place a proposal for approximately eight five-day preses- 
sions in the hands of a granting agency in July. 

Any AERA member interested in proposing and directing 
an AERA Pre session for 1969 should write the pre session 
Committee Chaiiman, W. James Popham, for a proposal out- 
line. Proposals are expected to be brief (no more than 
three or four pages) and tentative. There are no restric- 
tions on content; it is hoped that a broad range of topics 
(nonmethodological as well as methodological) will be pro- 
posed. The emphasis in the 1969 pre sessions will once 
more be training, i.e., providing participants with spe- 
cific competencies of relevance to their research activ- 
ities. The deadline for receipt of proposals is July 1, 

1968. The Presession Committee will meet shortly there- 
after to select those presessions deemed worthy of sup- 
port. 
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Request for proposal outlines and inquiries should he 

addressed to W. James Popham, Graduate School of Education, 

University of California, Los Angeles, California 9 0024. 

In addition, a somewhat more elaborate three -page duplicated call 
for presession proposals was mailed individually to approximately 200 
leading educational researchers in America. 

The Selection of Training Sessions . The Presessions Committee met 
at O’Hare Airport on Thursday, July 11, 1968* Douglas Penfield substi- 
tuted for Leonard Maras cuilo. All other committee members were present. 
The principal purpose of the meeting was to select a program of preses- 
sions in connection with the 1969 annual meeting. In addition, however, 
several decisions were made regarding extension of the presession-type 
program. 

The call for presession proposals published in the Educational Re- 
searcher plus personal notes from members of the Presessions Committee 
produced a total of 35 proposals. The titles of these proposals are 
listed below: 

1. Survey Research in Education 

2. Research in Instructional Product Development 

3. Laboratory Training in Educational Research 

4. Applying Research to Innovation in Higher Education 

5* Educational Research Management Training Program 

6. Language and Linguistics 

7* Evaluation: Information for Decision-Making - A Preview of 

the National Evaluation Study Committee of Phi Delta Kappa 

8. Audio-Visual Extensions of Research Techniques 

9* Methodological and Theoretical Problems of Research in Lower 
Income Communities 

10. Applied Linguistics in Educational Research 

11. Experimental Design in Educational Research 

12. Studying College Students 

13* Measurement and Evaluation: Test Construction 

l4. Multiple Group Discriminant Strategy for Multivariate Predic- 
tion of Taxonomic Criteria 
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15 . Nonparametric Methods and Associated Post Hoc Procedures in 
Educational Research 

16. Educational Information Handling 
17* Bayesian Statistical Analysis 

18. Design and Analysis of Comparative Experiments • 

19. Informal Instruction in Language Arts: A Challenge for Educa- 

tional Research. 

20. Use of Observational Systems 

21. The Computer and Natural Language 

22. Methods for Multivariate Data Analysis in Education and Psy- 
chology: Theory and Applications 

23. Training in Research on Elaborative Methods for Improving 
Learning Proficiency 

24. State of the Art of Remote Access Educational Systems 

25. Systems Approach in Counseling and Counselor Education 

26. Computer Applications to Education 

27. Introduction to Fortran IV 

28. Does Teacher Training Train Teachers? 

29. Fundament* .is of Educational Research 

30. Proposal Writing Workshop 

31. Biological and Social Detemi.nants of Ability 

32. Multivariate Design and Analysis in Educational Research 
33* Anthropological Methods in Education Research 

34. Mathematical Models for Measurement in Educational Research 
35* A Technology for Curriculum Development 

The meeting commenced with a reaffirmation of the statement of pre- 
session purposes which had been approved by the 1968 Pre sessions Com- 
mittee. The focus on training or disseminative sessions as opposed to 
seminal or generative sessions was particularly noted. It was agreed by 
the Committee that in judging the proposals of peers, since candor was 
important, any comments of an evaluative nature made during the meeting 
would be treated with complete confidentiality. 
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The first substantive decision pertained to the desirability of 
adding some type of training sessions on the East Coast. It was be- 
lieved that holding presessions in connection with the 1969 annual meet- 
ing in Los Angeles would make it difficult for Easterners, in particular, 
to attend. Eros and cons regarding the offering of East Coast sessions 
were deliberated at same length and a decision was finally reached that 
four sessions should be offered, but that they would follow the regular 
AERA meeting by a month or more. The decision regarding the designa- 
tion of which presessions to recommend for the East Coast was delayed 
until a consideration of the actual proposals. 

Turning to the major task of the meeting, that is, the evaluation 
of proposals, the following ten criteria were used by each evaluator: 

Director and Tentative Staff 

1 «. Experience and capability 

2 . Degree of commitment of tentative staff 

Content 

3. Importance of topic (need) 

4 . Appropriateness to presession format 

Instructional Objectives 

5 . Clarity 

6. Usefulness of competencies to be promoted 

Anticipated Audience 

7. Probable size 

Tentative Schedule 

"IT. Extent of planning 

Proposed Evaluation Activities 

9. Extent of planning 

10 . Comprehensiveness 

In addition, each evaluator gave the proposals an overall rating 
of either: 



A— reject 

B — accept conditionally 
C — accept with re commendations 
D— accept unconditionally 

All proposals had been previously received by mail, read, and evaluated 
by each committee member. Using the overall ratings at the outset, 
several initial rounds of evaluation were used to screen out those pro- 
porals considered inappropriate for one reason or another. The pro- 
posals surviving the first and subsequent elimination rounds were then 
discussed at considerable length. After several hours and the use of 
different ranking schemes, 12 proposals were selected as worthy of sup- 
port. The titles of the proposals which were selected and the names of 
the directors are listed below: 
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1. Research in Instructional Product Development 

Robert L. Baker 

2. Nonpar ame trie Methods and Associated Post Hoc Procedures in 
Educational. Research 

Leonard A* Maras cuilo 

3. The Computer and Natural Language 

Ellis B. Page 

If- Research on Methods for Improving Children* s Learning 
Proficiency 

William D. Rohwer, Jr. 

5. Systems Approach in Counseling and Counselor Education 

T. Antoinette Ryan 

6. Multivariate Design and Analysis in Educational Research 

Joe H. Ward, Jr. 

7. Anthropological Methods in Education Research 

Harry F. Wolcott 

8. Sample Free Test Calibration and Person Measurement in Educa- 
tional Research 

Benjamin D. Wright 

9* Survey Research in Education 

James G. Anderson 

10. Multiple Group Discriminant Strategy 

Paul R. Lohnes 

11. Bayesian Statistical Analysis 

Donald L. Meyer 

12. Design and Analysis of Comparative Experiments 

Kenneth Hopkins 

Jason Millman 

The procedure agreed to at the meeting was that the Committee chair- 
man should contact the 12 directors by phone as soon as possible and in=- 
foim them of the decision of the Committee. The 23 rejectees were sent 
airmail letters the following day. A personal letter was sent which 
described the Committee *s decision and the reason(s) for that decision. 

In addition, the submitter of the proposal was thanked for his expres- 
sion of interest. 

Having previously agreed to offer eight sessions on the West Coast 
and four on the East Coast, the Committee then recommended which ses- 
sions would be offered in the East and West. The chairman was given 
same discretion in modifying these preferences should certain directors 
be unwilling to hold their sessions in the East. 



The meeting was concluded with a discussion of particulars regard- 
ing the conduct of the 1969 sessions, call for 1970 proposals, etc* 
Prominent among those suggestions was the recommendation that an individ- 
ual “be recruited to handle logistical arrangements in Los Angeles for the 
presessions to he held there. It was also recommended that some kind of 
participant evaluation he conducted hy the Presessions Committee in addi- 
tion to whatever evaluations the session directors wished to conduct. A 
draft of an evaluation form was to he prepared hy the chairman who would 
send copies to each member for their modifications and suggestions. 

These forms were then to he taken to the various presessions hy the indi- 
vidual in charge of logistic arrangements and he transmitted directly to 
the Presessions Committee. Similar arrangements were to he made for 
evaluations of postsessions. Suggestions were also made regarding pre- 
session topics for next year. These suggestions were to he relayed to 
next year’s Presessions Committee chairman. 

The 12 session directors then revised their proposals so that they 
were incorporated into an AERA proposal to the U.S. Office of Education. 
This proposal was subsequently approved for funding hy USOE. 

Preparations for the Training Programs . A meeting on October 18 in 
Chicago was held with all 12 training session directors present. Discus- 
sions of the forthcoming pre- and postsessions centered around procedures 
for processing applications, in addition to a host of other procedural 
and substantive issues. The initial agreement concerned the method of 
processing applications. The decision of the group, after considerable 
deliberation, was to set no deadlines for applications, hut to judge ap- 
plicants in the order in which they were received on the basis of each 
applicant’s qualifications. In essence, the agreement reached was to 
employ a criterion-referenced rather than a noim-referenced scheme for 
accepting applicants. The step-by-step procedure for processing appli- 
cants to the 1969 training sessions is described below: 

PROCEDURES FOR PROCESSING APPLICANTS 
TO 1969 AERA TRAINING SESSIONS 

1. All applications received hy Popham immediately xeroxed and re- 
layed to first choice director. 

2. First choice director makes accept or reject decision within 
two or three days after receiving applications. (The very 
first few applications which trickle in can he retained some- 
what longer.) 

a. If an applicant is accepted, the director (l) sends the ac- 
ceptance letter to the applicant and (2) adds applicant’s 
name to a list of accepted participants which is sent 
weekly to Popham. 

h. If rejected, the application form is sent directly to sec- 
ond choice director (or, if no second choice is given, hack 
to Popham). 
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3- Second choice directors also make accept or reject decisions 
yith-m tvo or three days after receiving applications. 

a. If an acceptance is made, acceptance letter is sent by 
director, and applicants name is added to list being 
sent weekly to Popham. 

b. If rejected, return application form to Popham. 

4. Letters informing all rejected applicants will be sent by Popham. 
If certain directors wish to do so, they may send "sorry" let- 
ters also, but it is not necessary. Rejectees will be invited 
to apply to other sessions in which they are interested. 

5. Weekly totals of applicants accepted in all of the sessions will 
be mailed to each director. 

6. Directors for wham large numbers of participants present no in- 
structional problem are urged to accept as many qualified appli- 
cants as possible. 

It was also agreed to have a one page announcement of the training 
sessions sent to all AERA members as part of a forthcoming mailing 
from the AERA central office prior to the appearance of the November 
Educational Researcher (which was to contain the initial call for ap- 
plicants ) . 

The it ems required from each director for a final report regarding 
the training session were also determined. Further, procedures were 
described whereby anonymous evaluations from participants would be 
gathered on the final day of each training session. These were to be 
administered on a single page form by a representative of the Research 
Training Sessions Committee and returned directly to the chairman of 
that c ccmi ttee by the individual administering the form. Modifications 
in the one sheet evaluation form were made as a consequence of sugges- 
tions by the session directors. 



THE ERE SESSIONS 

Qjhg eight presessions were held in Los Angeles immediately prior ^o 
the annual meeting. A 25 percent random sample of the 384 applicants 
for these presessions revealed the following information: Eighty-two 

percent of the applicants were male and 18 percent female. The average 
age of the applicants was 36*5 years. Geographically, 43 percent of the 
applicants came from the West Coast (Alaska, Washington, Oregon, Cali- 
fornia, Nevada, Arizona and Hawaii) while 21 percent were from north- 
eastern states, 17 percent from midwestern states, eight percent from 

%his sampling plan was identical to that used in analyzing appli- 
cants to the 1968 presessions. Educational Researcher, No. 4, 1969* 
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southern states, and three percent from southwestern states. Canada 
contributed eight percent of the applicants. Of the individuals apply- 
ing for presessions, 22 percent had attended AERA presessions in previ- 
ous years. 

As expected, the majority of applicants (69 percent) were employed 
in college or university positions, while l4 percent were employed in 
public school systems, nine percent in federal government posts, four 
percent in state department of education positions, and four percent in 
other positions. 

Fifty-six percent of these applicants possessed the doctorate, 
while 4l percent held only a master* s degree. Three percent of the ap- 
plicants held neither a master *s nor doctoral degree. 

The research productivity of the applicants is reflected by an aver- 
age number of articles published in scholarly journals of 3*8. The ap- 
plicants had directed funded research projects to the extent of .8 pro- 
ject per applicant. 

The actual number of applicants (first choice) and participants in 
each of the pre sessions follows; 



Pre session Applicants 

1. Research in Instructional 
Product Development 

Robert L. Baker 68 

2. Nonparametric Methods and Asso- 
ciated Post Hoc Procedures in 
Educational Research 



Leonard A. Maras cuilo 42 

3. The Computer and Natural Language 

Ellis B. Page 55 

4. Research on Methods for Improving 
Children* s Learning Proficiency 

William D. Rohwer, Jr. 25 

5. Systems Approach in Counseling 
and Counselor Education 

T. Antoinette Ryan 31 

6c Multivariate Design and Analysis 
in Educational Research 

Joe H. Ward, Jr. 67 

(Continued) 



Participants 



rl. 



48* 

51 



18 

34 * 

58 



This disparity is accounted for by second-choice applications. 
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Presession 



Applicants 



Participants 



(Continued) 

7. Anthropological Methods in 
Education Research 

Harry F. Wolcott ^5 

8. Sample Free Test Calibration 
and Person Measurement in 
Educational Research 

Benjamin D. Wright 51 



Total 384 



THE POSTSESSiQWS 



The four postsessions were conducted after the annual meeting of 
AERA in College Park, Maryland, March 28 to April 1. A 25 percent ran- 
dom sample of the 274 applicants for the postsessions showed that 89 per- 
cent were male and 11 percent female. The average age was 35*8 years. 
Geographically, the greatest percent of applicants came from the north- 
eastern states which contributed 40 percent. Twenty-seven percent came 
from the upper midwestem states, and 24 percent from the southern 
states. Canada contributed four percent while the West Coast, south- 
western and midwestern states contributed approximately two percent each. 
Of the applicants for the postsessions, 29 percent had previously at- 
tended AERA presessions. 

As with the presession applicants, the sample for the postsessions 
showed the majority of applicants to be employed in colleges or univer- 
sities (74 percent). Six percent were employed in public school systems, 
10 percent in federal government posts, three percent in state department 
of education positions, and seven percent in other positions. 

Seventy— four percent of the applicants possessed the doctorate, 23 
percent held the master’s degree, and three percent held neither. In- 
formation on research productivity revealed that the applicants had an 
average of 3.8 published articles in scholarly journals, and had di- 
rected an average of 1.2 funded research projects. 

The actual number of applicants (first -choice) and participants in 
each of the postsessions follows: 



Postsession 


Applicants 


Participants 


Survey Research in Education 
James G. Anderson 


96 


63 


Multiple Group Discriminant Strategy 
Paul R. Lohnes 


51 


52 


Bayesian Statistical Analysis 
Donald L. Meyer 


42 


31 


Design and Analysis of Comparative 
Expei'iments 

Kenneth Hcpkins 
Jason Mill man 


85 


56 


Total 


274 


202 



ANONYMOUS EVALUATIONS OF THE TRAINING SESSIONS 

For the first time in the history of the AERA training sessions a 
standard evaluation form was administered anonymously at the close of 
each presession and postsession by representatives of the 1969 AERA Re- 
search Training Sessions Committee. These one page evaluations con- 
tained the following directions: 

This evaluation form is administered directly by the 1969 
AERA Research Training Sessions Committee. Completed forms 
will be returned directly to the Committee by the person ad- 
ministering this form. After the data have been tabulated, 
the instructional staff of your session may request a summary. 

The principal purpose of this form is to assist in the plan- 
ning of next year*s training sessions. Therefore, be com- 
pletely candid in your responses. Do not sign your name. 

The participants* evaluations of each session conclude the individual 
description of that session in subsequent sections of this report. An 
examination of those evaluations will indicate that all pre- and post- 
sessions were viewed positively by participants. Administration of the 
form took only the expected five minutes. However, particularly with 
the presessions (which were held in several different locations), the in- 
dividuals administering the forms had to be in several parts of the Los 
Angeles area within a few hours. As a consequence, less than 100 per- 
cent of the participants completed the evaluation forms in some cases - 
due to conflicts with coffee breaks, early departures, etc. 



To provide an overall estimate of the quality of the sessions, as 
reflected by participants* anonymous evaluations, in Table 1 those 
responses which pertain to the effectiveness of a session are summarized 
for presessions, postsessions, and for both combined. The responses are 
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presented in percentage form, following the actual question as it ap- 
peared in the evaluation form* 

Table 1 

Percentage S umma ries of Participants* Anonymous Evaluations 
of Presessions, Postsessions, and Both Combined 



Question: "Please rate the overall quality of instruction in your 

Excellent Good Average Fair Poor 



se ssion- 



Presessions ■* 


56 


32 


6 


Postsessions # 


45 


40 


9 


Combined ** 


52 


35 


7 



4 

3 

4 



2 

3 

2 



Question: 



"Leaving aside the quality of instruction for the moment, do 
you think the topic treated in your session should be treated 
again next year?" " 





Yes 


HT- 
11 0 


Uncertain 


Pre sessions * 


87 


3 


10 


Postsessions # 


94 


4 


2 


Combined ** 


90 


3 


7 


Question: "If you had it 


to do over 


again, would you apply for the ses- 


sion which you 


have just completed? 






Yes 


No 


Uncertain 


Presessions * 


87 


8 


5 


Post sessions # 


81 


9 


10 


Combined ** 


85 


8 


7 


Question: "If a session 


such as this 


; is held again. 


would you recommend 


to others like you that they attend? 






Yes 


No 


Uncertain 


Pre sessions * 


90 


4 


6 


Post sessions # 


88 


7 


5 


Combined ** 


89 


5 


6 


* n = 250, # n = 153, ** 


n = 403. 







An examination of these summary percentages reveals a very favorable 
participant response to the overall program. There appear to be only 
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minor differences in participant responses to the presession programs, 
as opposed to the post session programs* It seems that approximately 
90 percent of the 250 presession participants and 153 postsession par- 
ticipants ■who completed the anonymous evaluation foim were well satis- 
fied with the programs. 



COST TO PARTICIPANTS 

The anonymous form completed "by participants on the concluding day 
of each session also solicited information regarding how much money, 
both re imb ursable and nonreimbursable, the training session had cost 
the participant. The presession and postsession median figures for 
both these costs are presented in Table 2. 

One should note that in connection with the postsessions a $75 
stipend was provided participants, if needed, by the USOE contract 
funds supporting the sessions. Since the presessions were held in jux- 
taposition to the annual AERA meeting, a stipend was not considered 
requisite. Using average estimates of 180 reimbursable dollars and 
70 nonreimbursable dollars per presession participant multiplied by 
339 participants, one calculates a hard dollar nonfederal contribu- 
tion of approximately $85*000 for the presessions. For the postses- 
sions, using a $100 reimbursable figure, less the USOE $75 reimburse- 
ment - or $25, plus a nonreimbursable estimate of $60, the nonfederal 
postsession contribution approximates $17,000* The combined total of 
over $100,000, of course, represents a considerable hard dollar invest- 
ment on the part of the educational research community in these train- 
ing efforts. But one must add the soft dollar costs such as those 
borne by the agencies and institutions which usually permit their pro- 
fessional staff members to avoid regular responsibilities for several 
days while attending one of these technical refurbishing sessions. Or, 
to cite another less apparent cost, consider the nonreimbursed time 
spent by the AERA committees involved in setting up and administering 
these sessions. When one adds these more elusive costs to the hard 
dollar estimates, the nonfederal financial contributions to the AERA 
training sessions are indeed considerable. 



Table 2 

Median Reimbursable and Nonreimbursable Costs per Participant 



Session 


Reimbursable 


Nonre imbur sable 


Presession I 


$197 


$57 

34 


Pre session II 


217 


Presession III 


210 


52 



(Continued) 
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Table 2 



Median Reimbursable and Nonreimbursable Costs per Participant 



Session 


Reimbursable 


Nonreimbursable 


(Continued) 






Presession IV 


$125 


$ 83 

_l 


Pre session V 


13^ 


74 


Pre session VI 


178 


51 


Pre session VII 


154 


123 


Presession VIII 


220 


71 


Postsession I 


91 


72 


Postsession II 


99 


47 


Postsession III 


120 


45 


Postsession IV 


101 


58 


PUBLICITY EFIECTIVENESS 



As part of the final day*s anonymous evaluation forms participants 
were also asked to indicate where they first learned about the training 
session they had just completed. These data were needed in order to 
gauge the effectiveness of the several promotion schemes which were em- 
ployed by the AERA central office staff and by members of the 1969 Re- 
search Train ing Sessions Committee. In Table 3 results for each ses- 
sion are presented: 

Table 

Participants* Identification of 
First Information Source Regarding 
The Training Session (In Percentages) 




Educat ional Professional One Page AERA Other 

Session Researcher Journal Announcement 



Presession I 


71 


7 


4 


Pre session II 


76 


4 


0 


Pre session III 


46 


18 


7 


Pre session IV 


64 


14 


7 


Presession V 


33 


13 


13 


Presession VI 


75 


4 


5 


Pre session VII 


70 


4 


11 


Presession VIII 


62 


0 


8 



(Continued) 



18 

20 

28 

14 
4o 
16 

15 
30 
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Table ^ 



Participants* Identification of 
First Information Source Regarding 
The Training Session (in Percentages) 



Session 


Educational 

Researcher 


Professional 

Journal 


One Page AERA 
Announcement 


Other 


(Continued) 
Postsession I 


69 


11 


9 


11 


Postsession II 


76 


3 


14 


7 


Postsession III 


70 


0 


19 


11 


Post session IV 


72 


4 


12 


12 



The majority of "other” sources included information from colleagues, 
letters from session directors or staff, and informal newsletters from 
professional subgroups. It should be noted that information regarding 
two of the sessions was communicated to a majority of participants by 
vehicles other than the one page AERA announcement and the Educational 
Researcher . Apparently, for certain sessions diverse promotional 
schemes can be more effective than the customary channels. 



SUMMARY OF INTRODUCTORY REMAR KS 

Emerging from a widely voiced need by the educational research com- 
munity for additional training, the 1969 AERA presessions and postses- 
sions continued the trend established in the mid- sixties for short-term 
training institutes directed by senior researchers. For the first time 
a program of postsessions was offered in addition to the customary pre- 
sessions. In all, the eight presessions and four postsessions attracted 
658 applicants, 539 of whom actually participated in the sessions. Both 
the evaluation data supplied by individual directors (see next section) 
as well as participants* anonymous responses to a form administered 
directly by the 1969 AERA Research Training Sessions Committee indicate 
that the sessions were extremely successful. 

In the remainder of this report descriptions of the 12 training 
sessions are presented with each of the following elements being in- 
cluded : 



1. Title 

2. Staff 

3. General Description 

4. Objectives 

5. Schedule 

6. Participants 
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7 . Instructional and Evaluation Materials 

8. Evaluation 



a. Test Results 

b. Director’s Evaluation 

These descriptions are largely drawn from reports supplied by the direc- 
tors of the 12 sessions. Minor editing wan undertaken to preserve some 
degree of uniformity in the reports. The final section of each report, 
that is, the participants* anonymous evaluation of the session, was added 
to the information supplied by the director. 



TRAINING SESSION DESCRIPTIONS 




o 



PRESESSION I: RESEARCH IN 

INSTRUCTIONAL PRODUCT DEVELOPMENT 



Staff 



Robert L. Baker 
(Director) 

Harry Handler 
William Hein 



W. James Popham 



Richard E. Schutz 



Southwest Regional Laboratory for 
Educational Research and Development 

Southwest Regional Laboratory for 
Educational Research and Development 

Southwest Regional Laboratory for 
Educational Research and Development 

University of California 
Los Angeles, California 

Southwest Regional Laboratory for 
Educational Research and Development 

Southwest Regional Laboratory for 
Educational Research and Development 



General Description 

The focus of this presession was on the research operations asso- 
ciated with the systematic development of instructional products (i.e., 
materials). Product research activities at the Southwest Regional Lab- 
oratory for Educational Research and Development have led to the prep- 
aration of a series of instructional sequences designed to promote the 

skills necessary to initiate and conduct product research and develop- 
ment. 



The presession was not an academic offering as such. Activities 
were generated around selected SWRL and UCLA developed materials re- 
lated to seven instructional objectives. Experience with these objec- 
tives has demonstrated dramatically that behavioral change within a 
five-day p>eriod may be directional but never complete. In view of the 
limitations imposed, the general strategy was to give the concepts, is- 
sues, and procedures sufficient coverage during the presession to enable 
the participant to accomplish the stated objectives as a function of 
real-time practical efforts in his own laboratory. 

Objectives 

Participants were to be able to: 

Objective 1: Specify in behavioral terms the desired outcomes of 

an instructional program. 

Objective 2: Describe the components of a set of instructional 

specifications and construct sample specifications 
related to stated objectives. 
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Objective 3* Select, describe, and/or construct the components of 
valid instruments to measure the extent to which the 
desired outcomes are attained. 

Objective 4: Describe a given learning situation in terns of its 

essential elements and outline effective strategies 
for developing instructional materials appropriate 
for defined functions. 



Objective 5: Select the most valid and practical experimental de- 

sign for investigating the specified relationships. 

Objective 6: Identify the essential ingredients of a written sum- 

~~ mary and evaluation of an entire development cycle 
with a particular product. 



Schedule 

The institute scheduled activities from 8:30 a.m. to 11:45 a.m. and 
1:15 p.m. to 4:45 p.m. In general, there were three "formal" sessions 
per day and two "optional" sessions. The evening sessions were always 
optional and were directed to in-depth treatment of topics in which 
participants displayed interest. 



The instructional program and materials were organized to provide 
considerable discussion on each topic and objective. The rather heavy 
reading requirements necessitated strong encouragement that partici- 
pants schedule regular evening hours for independent study. One ses- 
sion each day was developed to direct instruction and practice on the 
previously assigned instructional materials. Each session was staffed 
by more than one instructor; however, one staff member was assigned the 
principal responsibility for organizing and conducting each session. 



2SE I 



Session 1: 
Session 2: 
Session 3: 
Session 4: 
Session 5: 



Organization, Pretest 
Product Development Research 
Product Development Research 
Elective Activities, Session Organization 
Issues in Product Research, Design, and Develop- 
ment 



Day II 



Session 1; 
Session 2: 

Session 3: 
Session 4: 
Session 5: 



Instructional Objectives 

Constructing Objectives and Instructional Specifi- 
cations 

Criterion Measures and Instructional Decisions 
Elective Activity 

A Further Look at Instructional Objectives 
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Day III 



Session 1: 
Session 2: 
Session 3- 
Session 4: 
Session 5; 

Day IV 

Session 1: 
Session 2: 
Session 3: 
Session 4: 
Session 5; 

Day V 

Session 1: 
Session 2: 
Session 3: 
Session 4: 

Participants 



Product Development Strategies and Procedures 
Product Development Strategies and Procedures 
Quality Verification Procedures 
Elective Activity 

Product Tryout and Revision Strategies 



Management Procedures in Product Development 
Costing and Program Budgeting Strategies 
Research Design, and Development Strategies 
Elective Activity 
Design Issues 



Research in Product Development: A Summary 

Posttest 

Session Evaluation 
Posttest feedback and wrap-up 



Approximately two-thirds of the 54 participants were from college or 
university staffs. Less than one-fifth were from the public schools. As 
a group they represented a sophisticated professional education level, 
having published an average of seven journal articles. Sixty percent of 
the participants had at least one funded project for which they were re- 
sponsible. Sixty-three percent of the participants possessed a doctorate. 

Sixty-nine percent of the participants were male and 31 percent fe- 
male. The group* s average age was 38 years. 

Materials 



In all , 38 documents related to eight content areas were distributed 
to the participants. The eight content areas were as follows: (l) Spec- 

ification of Outcomes, (2) Instructional Specifications, ( 3 ) Measurement 
of Outcomes, (4) Development of Instructional Materials, (5) Research De- 
signs, (6) Product Development Management, ( 7 ) Program Costing and Budget- 
ing, and (8) Educational Research and Development. 

Thirteen of the documents were " self-contained" instructional se- 
quences which the participants worked through during non-scheduled class 
hours. An attempt was made to provide maximum cues to the participants 
for use of the materials when they returned to their respective home 
bases. 
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Evaluation 



Test Results , The primary sources used to evaluate the presession 
were the pre- and post-instructional performances of the participants on 
the instructional objectives. The two tests used were neither parallel 
nor equivalent. The pretest had only 67 items; the posttest had 115* 

An attempt was made to sample from the same domain for both tests. To 
avoid pretest "frustrations, " part of the sampling rationale for the 
posttest was to include the more difficult items which participants 
would probably not be able to respond to at the outset of instruction. 
Although this makes direct camparisions somewhat questionable, differ- 
ences obtained would be biased toward the lower bounds. 

Table 4 shows the Group Profile of median scores for the pretest 
and posttest, by objective. 



Table 4 

* Group Profile: Median Scores (percentage) for Pre- and Posttest, by 

Objective. 





Objective 


Content 


Percentage 


Percent Correct 







Pre 


Post 










2( 


) 3C 


) 4C 


) 5< 


J 60 70 80 


90 


1 


Behavioral Objectives 


69 


81 








1 *1 1 
» t « / 




2 


Instructional Specifica- 


64 


72 








t £ 1 A 

1 \ I \ 1 
It* \ 1 
1 . 1 \l 
1*1 % 
1 I 

1 VI 1 






tions 














3 


Selection of Criterion 


69 


78 












Tests 












I 4 ft 
1 l\ 








76 


84 








1 »> « \ 




4 


Developing Materials 








I A l 1 

jT I I 




5 


Design/ Evaluat ion 


46 


59 








' 1/^ • I 

1 1 1 





*Dotted line related to pretest results; solid relates to posttest 
results. 

Obviously, the "gains" on the posttest were modest. Two things 
contributed to this. One, the posttest was a more difficult test than 
the pretest. Two, the heavy reading load with little "teeth" in the 
mastery-level monitoring led to insufficient control over concept de- 
velopment. During the five-day period much emphasis was placed on the 
participant monitoring his own progress and determining the functional 
utility of the compendium of material for his own professional situa- 
tion. 
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Following the presession two additional instruments were used to 
help evaluate the overall effectiveness and acceptance of the presession. 
The first of these was the AERA 1969 Research Training Sessions Partici- 
pant Evaluation Form administered by the AERA central headquarters staff. 
The form was administered with complete anoi^mity of the responder. If 
we can accept such a self-report of global presession characteristics as 
at least partial evidence of success, the results (reported later) defi- 
nitely suggested that there was "customer acceptance." 

In addition, a 55 forced choice dyad device was constructed to de- 
termine priorities as established by the participants. They were asked 
to rate the topics in each of the 55 dyads in terms of their comparative 
importance to them. Table 5 includes a summary of the results. Since 
each of the topic areas was paired once with every other topic and thus 
was included in 11 dyads, the percentage of times each was selected re- 
fers to the base-eleven. The results indicate that Evaluation Strategies 
was perceived as being most important and Program Costing and Budgeting 
least important. A certain amount of conceptual confusion is evidenced 
by the fact that although Evaluation Strategies is ranked high. Quality 
Verification Procedures is very modestly rated. 



Table 5 



Priorities Questionnaire Results 



♦Percentage of 
Times Selected 


Topic Area 


69-37 


Evaluation Strategies 


63.33 


Developing Instructional Specifications 


60.00 


Criterion Test Construction 


58.33 


Developing Instructional Materials 


57.50 


Constructing Objectives 


55.00 


Tryout and Revision Procedures 


47.50 


Quality Verification Procedures 


44.37 


Defining Treatment Variables 


42.08 


Research Design and Statistics 


34.37 


Research Management Procedures 


18.12 


Program Costing and Budgeting 



*100# would indicate that the topic was the first choice in every dyad 
in which it appeared. 



Directors Evaluation . The overall evaluation of the presession 
was that it was quite worthwhile. Recognizing that the objectives 
stated are not amenable to mastery within a five-day period, even when 
the participating group is homogeneous and "ready," the presession 
staff was faced with two general alternatives. One, we could have re- 
stricted the content and objectives to a manageable five-day program. 
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tiSU C °^ trUCtlnS objectives 804 designing instructional specifica- 
t °" s ‘ we could have provided a broad survey of the whole of 

^oduct developnent, handling the concepts rather academical and at 

„ +nf . r y a , Te f ba ^ leve1, Neither alternative was satisfactory to the 
to8lys *f °f our clientele suggested that the former was too 
anv^real fn P*°cedurally, while the later did not permit 

“L!?? 1 £ t ^ ccmprcmise was based on the assumption that a de- 
pendabie behavior change was not likely irrespective which route we 

took. Our general strategy was predicated on the hope that the oartic 

lT4i:^“"edT tt deVel ° P SkiUS ^ W ° rk ^the lteriSf 

when they returned to their own professional home. The obsession cur 

2 COVS * the entire ran « e «* coSeX^S Sject^s 

troduc tjon- Attention was then directed to helping the 

tained^nstructiona? 38 pieC6S ° f Written ^^rials both as self-con- 
t ained instructional, sequences and as ready references. If we could 

riSa ifa : “"“f ^ °™ ^formance in working throu^ the mte- 

izUbilit^t^tte^ Bt thS ; S iU laTel ’ ^ instructional sane cal - 

inability to their own professional effort at the real-time work level! 

^rticipants* Anonymous Evaluations . On the final day of the ses- 
particrpants completed an anonymous evaluation fom which was 
administered by and returned directly to the 1 969 AERA Research Tra-'n 

A vu- 1- If y0U ha<i to do over -would you apply for the ses- 

s on which you have just completed? Yes 8h%, No %, Uncertain 7%. (k) 

If a session such as this is held again, would you reccranend to others ^ 
like you that they attend? Yes 89$, No % } Uncertain k%. 
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IRE SESSION II: NO NPARAMETRIC METHODS AND 

BELATED POST HOC PROCEDURES IN EDUCATIONAL RESEARCH 



Staff 



Leonard A. Marascuilo 
(Director) 



University of California 
Berkeley, California 



Maryellen McSweeney 



Michigan State University 
East Lansing, Michigan 



Douglas Penfield 



University of California 
Berkeley, California 



Andrew Porter 



Michigan State University 
East Lansing, Michigan 



Michael Subkoviak 



State University of New York 
Buffalo, New York 



General Description 

The course content of this session was restricted to nonparametric 
techniques extensively used in education and the behavioral sciences. 

The first part of the course emphasized tests designed to handle quali- 
tative variables. The analysis of both large and small samples was dis- 
cussed. The second part of the course emphasized nonparametric tests 
based on ranking procedures. Confidence procedures useful in establish- 
ing bounds on the parameters upon rejection of the hypothesis under test 
were discussed in detail. Measures of association for qualitative and 
ranked variables were touched upon. The primary objective was to assist 
the participants in applying the techniques of nonparametric statistics 
to the solution of problems of hypothesis testing and estimation commonly 
occurring in educational research. To accomplish this goal, the lec- 
tures emphasized (l) development of an "intuitive” rationale for the 
procedures and (2) applying the procedures in many illustrative research 
contexts. The mathematical development of the various tests discussed 
were not covered as part of the formal lectures. 

This session was op>en to persons whose responsibilities included 
the design of educational research studies and the analysis of research 
data. The course was intended for educational researchers whose primary 
ccmmitment is to substantive areas other than that of statistics. 

Objectives 

The primary objective of the presession was to assist the partic- 
ipants in applying the techniques of nonparametric statistics to the 
solution of problems of hypothesis testing and est ima tion commonly 
occuring in educational research. 
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Participants, at the close of the session, should have been able 

to: 

Objective 1: Understand the rationale behind the nonparametic 

tests presented. 

Objective 2: Select the “best" nonparametric test for a specific 

situation. 



Objective Perform nonparametric tests on data from educational 
research. 



Objective 4: 



Conduct post hoc comparisons on hypotheses rejected 
by nonparametric procedures. 



Objective Apply and extend the techniques learned to specific 
problems in educational research. 

Objective 6: Read current literature on nonparametric methods ap- 

propriate for research in education. 



Objective 7: Direct other researchers in the use of nonparametric 

statistics. 



Schedule 



Day I 

Session 1: 

Session 2: 
Session 3: 
Session 4: 
Session 5: 

Day II 

Session 1: 
Session 2: 
Session 3: 

Session 4: 
Session 5: 

Day III 

Session 1: 

Session 2: 

Session 3* 



Discussion of class objectives and review of clas- 
sical statistical methods 
Continuation of review 

The Binomial and Hypergeometric Distribution 

Fisher exact test 

Discussion 



Median test and Chi-square test 

Extension of the Median test to K samples 

Post hoc procedures for the Chi-square and Median 

tests 

Tests for Independence in Contingency tables 
Discussion 



Tests of interaction across contingency tables 
and associated post hoc procedures 

X 2 tests for interaction in multivariable contin- 
gency tables 
The sign test 
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Session 4: 

Session 5? 

Day IV 

Session Is 
Session 2: 

Session 3* 

Session 4: 
Session z>\ 

Day V 

Session 1: 
Session 2: 

Session 3: 
Session 4: 

Session 5: 



Spearman* s rho and the Wilcoxon test for matched 
pairs 

Discussion 



Friedman test 

Post hoc procedures for the Friedman test and 
Kendall* s Coefficient of Concordance 
Cochran* s Q. test and associated post hoc proce 
dures 

Kendall *s tau 
Discussion 



Wilcoxon test (Mann-Whitney) 

Kruskal-Wallis test and associated post hoc proce 
dures 

Normal scores test 

Application of the Wilcoxon test to block designs 

and test on aligned observations 

Discussion 



Participant s 

Of the 48 participants admitted to the presession, 34 appeared on 
the opening date. Thirty remained with the presession throughout its 
duration. As might have been expected, the bulk of the participants were 
male, and produced about the same distribution as in the 1968 pre session 
on the same topic: seventy-three percent were male and 27 percent female. 

Most of the participants were under forty years of age. Fifty-four per- 
cent were from a college or university, 23 percent from a research center, 
and 23 percent from other areas. Most of the participants held profes- 
sorial positions at their respective teaching institutions. For most of 
the participants (62 percent), this was a first exposure to AERA preses- 
sions. Seventy-nine percent held a doctorate, and more than 40 percent 
had at least one article accepted for publication in a refereed journal. 
The average number of funded research projects directed by each partici- 
pant was 1.2. 

Approximately 28 percent of the applicants were from the West. The 
very large number of non-show individuals were from the East and Middle 
West. Most did not attend because of financial cuts and the shortage of 
travel funds. 

Materials 



East staff member prepared detailed lectures. These lectures were 
presented and given to the students. Participants could refer to the 
printed materials as the instructor lectured and, therefore, did not have 
to write notes or copy from the board. 
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The first part of the course consisted of a review of the "basic 
parametric tests generally discussed in a two -quarter or two -semester 
course on statistics. A printed review of this material was mailed to 
participants a month prior to the time of the presession. The second 
part of the course treated nonparametric tests designed for qualitative 
variables, and the third part of the course emphasized nonparametric 
tests based upon ranking procedures. The mathematical development and 
proving of theorems and related results appeared in the handout materi- 
als. The prepared handouts accompanied each lecture. Related readings 
were assigned in the text Distribution-Free Statistical Tests by James 
V. Bradley, WADD Technical Report, 6o-66l (Office of Technical Ser- 
vices, U.S. Department of Commerce, Washington, D.C.). 

Evaluation 



Test Results . Each student was given five multiple- choice tests 
containing very difficult items. The statistics for the first three 
tests are as follows: 



Test 



Median Maximum 

No. Correct Score Possible 



One 7 10 

Two 15 20 

Three 10 20 



The low scores on the test were not unusual nor were they un- 
expected by the staff. While knowledge of the binomial and hyper- 
geometric distributions is essential to the use of many nonparametric 
methods, it turned out that most participants had only superficial 
knowledge of these important statistical distributions. Another weak- 
ness noted by the staff was the relatively poor training that students 
in schools of education obtain concerning the importance of confidence 
intervals. It seems that most instructors of statistics overemphasize 
the testing of null hypotheses and ignore the more important procedures 
associated with confidence intervals and their use for making statis- 
tical inferences. Since the presession emphasized the use of simulta- 
neous multiple confidence interval procedures, almost all of the mate- 
rial presented to the participants was new. 

Director* s Evaluation . The presession on Nonparametric Methods 
and AssociatecMPost Hoc Procedures was held at the Marina Del Rey Hotel 
on the five days just prior to the 1969 AERA Convention in Los Angeles. 
While most of the participants were unprepared for the course planned 
by the instructional staff, the number of new ideas and the intellectual 
growth that took place in the participants clearly shows that the pre- 
session was successful. In fact, the staff believes it was more suc- 
cessful than the 1968 presentation. While the presession did not 
satisfy all of its originally stated objectives, it did open new vistas 
and new ways of looking at data collected in educational settings. The 
value in conducting this presession will be measured in the payoff to 
education. Both the staff and the participants expect this payoff to 
be large. 
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Participants * Anonymous Evaluations . On the final day of the ses- 
sion 25 participants completed an anonymous evaluation form which was 
administered by and returned directly to the 19^9 AERA Research Training 
Sessions Consnittee. Four items on that form pertained directly to the 
quality of the session. Responses to these questions follow: 

(l) Please rate the quality of instruction in your session. Excellent 
76$, Good 8$, Average 4$, Fair 8$, Poor 4$. (2) leaving aside the 

quality of instruction for the moment, do you think the topic treated 
in your session should be treated again next year? Yes 100$, No 0$, 
Uncertain 0$. ( 3 ) If you had it to do over again, would you apply for 
the session Sfhich you have just completed? Yes 9 6$, No 4$, Uncertain 0$. 
(4) If a session such as this is held again, would you recommend to 
others like you that they attend? Yes 9§$> No 4$, Uncertain 0$. 



PRE SESSION III: THE COMPUTER 

AND NATURAL LANGUAGE 



Staff 



Ellis B. Page 
(Director) 


University of Connecticut 
Storrs, Connecticut 


Dieter H. Paulus 
(Assistant Director) 


University of Connecticut 
Storrs, Connecticut 


Harold Borko 


University of California 
Los Angeles, California 


Carl Helm 


City University of New York 
New York, N.Y. 


Bruce Fraser 


Director of the Lanaguage Research 
Foundation 

Cambridge, Massachusetts 


L. Stephen Coles 


Stanford Research Institute 
Stanford, California 


John McManus 


Southwest Regional Laboratory for 
Educational Research and 
Development 



General Description 

Since ancient times, the main line of Western scholarship has cen- 
tered around the analysis, transformation, and production of written 
language. Until very recently, however, such fields have lacked the 
tools and techniques and concepts necessary for large-scale, reliable, 
and inexpensive analysis of language $ and high-level scientific research 
has, therefore, been nearly impossible. But recent developments in com- 
puter science make possible a revolution in verbal scholarship and sug- 
gest radical changes in the scope and depth of certain kinds of educa- 
tional research and practice. The content of the presession was con- 
cerned with the techniques and applications of such research. Tech- 
niques included certain higher-level language (especially FORTRAN, PL/ I, 
and list-processing languages), syntactical and other analysis, and 
statistical optimization. Applications will include essay grading, con- 
tent analysis, decision-making, information retrieval, computer-assisted 
instruction, research in the humanities and in guidance. There was 
active involvement with conversational computer terminals > 

Objectives 

The eventual outcome expected was that the participants would do 
research using natural-language analysis, would be able to help 
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researchers in the softer areas perform such analysis, and would teach 
such research techniques to students and faculty in their own regions 
and institutions. In this way, an effective and virtually new area of 
educational research might he opened for active exploration and employ- 
ment. 



The four basic goals of the session were that participants would 
be able to; 



Objective 1: Learn about list processing languages, and to master 

actual programming techniques, which are especially 
suited for natural language programming. 



Objective 2 : Learn strategies of heuristic programming, psycho- 

logical simulation, semantic memory, and artificial 
intelligence appropriate for such analysis. 



Objective Learn techniques of essay grading by computer, con- 
tent analysis, and computational linguistics which 
may illuminate certain educational problems. 

Objective 4: Conceive, flow-chart, and partially program same 

suitable sma l l strategies for possible later in- 
corporation into larger educational researches. 



Schedule 



Dajr I 



Session 1; 



Session 2: 



Session 3: 



Session 4; 



Introduction; Views of Natural Language; general 
concepts, statistical approaches, analytical ap- 
proaches 

Natural Language Techniques; principles and con- 
veniences, using FORTRAN for natural-language 
programming, alternative languages 
interdisciplinary Structure of the Work; parent 
societies, publications, major centers and proj- 
ects 

Special interest seminars and groups 



Day II 



Session 1; 
Session 2; 



Session 3: 
Session 4; 



Language and Logic 

Applications to Humanities; categories of mea- 
surement, information retrieval, stylistic anal- 
ysis, comparisons and trends, steps toward crit- 
icism, status of the field 
Special interest seminars and groups 
Special interest seminars and groups 



Day III 

Session 1: 

Session 2: 

Session 3: 
Session 4: 
Session 5 : 

Day IV 

Session 1: 

Session 2: 

Session 3s 
Session 4: 

Day V 

Session 1: 
Session 2: 



Applied language Analysis: Deterministic Proce- 

dures 

Applied Language Analysis: Statistical Proce- 

dures 

Language Data Structures: Information Retrieval 

Question Answering and Decision-Malting 
Voluntary workshops or interest groups; Beginning 
own construction of proposed research in natural 
language analysis; Working with computer terminals 
Reading, etc. 



Artificial intelligence: search, pattern recogni- 

tion and relevance to NLA, learning systems and 
the improvement of analysis, problem- solving and 
planning, induction and models for future analysis 
Same Educational Applications: interaction in in- 

struction and guidance, educational languages, use 
of information retrieval in guidance, computer 
aided instruction, other applications such as 
creativity tests 

SmajJL groups and individual problems; Work with 
the computer terminals 

.Qwflli group and individual work; Terminal and 
reading, etc. 



Available Resources and Further Problems: avail- 

able facilities, 1f What We Should Be Working On*' 
Summary and Evaluation of the Presession 



Participants 

Of the 51 participants 88 percent were male and 12 percent female. 
Their average age was 31. 9. Information from the participants revealed 
that 50 percent held a doctorate and the average number of research 
articles accepted in a scholarly journal was 4.7 per participant. 



Materials 



The staff of the presession set up a library of 113 volumes and 
periodicals relating to natural language computing. These materials 
were available during most of the pre session to all participants who 
were allowed to check them out overnight. 

Also, each staff member brought with him a selection of materials 
to supplement his lectures. These materials served either as explana- 
tory, more detailed materials, or as helpful outlines for the content 
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of narticular lectures. 



Examples of the types of instruction and evaluation materials used 
were: Exercise II-A-3, Complete the Following Program (sample program 

and assignment of string manipulation program written in FORTRAN IV^ 1 
page); Content Analysis - The General Inquirer (brief outline describ- 
ing structure and function of the General Inquirer System, 3 P a S es ) i 
ELEVR SCRIPT (script illustrating the ELIZA system, 1 page); Puckett, 
STUFF (description of and directions for using the STUFF system, . 3.4 
pages); Review on Computer Grading of Essays (seven review questions 
on Project Essay Grade, 2 pages); Instrument I-C (thirty item ques- 
tionnaire on semantic differential structure, 2 pages); and Quiz on 
Natural Language Application to Education (twenty item quiz on natural 
language computing applications to education, 1 page; . 



Evaluation 



Test Results. Many evaluative devices were designed to serve two 
relatively distinct purposes. First, the many content mastery tests, 
designed, administered and scored by the various staff members about 
the quality of their instruction. These instruments were generally 
administered immediately following each lecture. In many cases the 
correct responses tc each item were also distributed, so as to provide 
feedback to the participants. 

Another class of instruments was designed to evaluate the preses- 
sion as a whole. Some of these devices were administered on the first 
day of the presession as “pretest" instruments. Others, including 
open-ended questionnaires, were administered and scored after the sec- 
ond complete day of the presession. This was done so that the preses- 
sion organization and/or procedures could be changed if needed. Fi- 
nally, instruments were administered at the close of the presession. 
These were post-tests, corresponding in some cases to the pretests 
given earlier. 

Instrument II-B, Formative Evaluation Questionnaire, was admin- 
istered twice during the presession. The first administration, after 
the second day of the meeting, provided information about the general 
satisfaction of the participants with the way in which the presession 
was conducted. The second administration, at the end of the pre session, 
yielded similar information for the entire meeting. Results are pre- 
sented in Table 6. As can be seen by the results reported on the in- 
struments, the participants were quite happy with the way the meeting 
was conducted. The results of the second administration indicate that 
this general satisfaction increased throughout the meeting. 
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Table 6 

Instrument II-B: Fomnative Evaluation 

Questionnaire Results 



Key: SA (Strongly Agree), A (Agree), (Undecided), D (Disagree), 

SD (Strongly Disagree")", KA (No Answer). Please circle your 
choices. 

Second Day Fifth Day 



8 . 



SA A ? D SD NA j SA A ? D SD M 



1. The objectives of 
this program were 
clear to me. 

2. The objectives of 
this program were 
not realistic 

3. The participants 
accepted the pur- 
poses of this 
program. 

4. The objectives of 
this program were 
not the same as my 
objectives. 



9 17 9 7 0 0 



1 13 7 15 6 0 



4 21 17 0 0 0 



0 13 8 15 6 0 



5. I have not learned 

much new. 2 9 5 15 H 0 

6. The material pre- 

sented seemed valu- 
able to me. 10 20 9 3 0 0 



I could have 
learned as much by 
reading a book. 

Possible solutions 
to my problems are 
not being con- 
sidered. 



2 13 7 15 5 0 



9. The information 
presented was 
too elementary. 



2 10 18 7 3 2 



2 6 7 19 6 2 



10. The speakers really 
know their sub- 
jects. 16 25 0 0 0 1 



9 22 3 6 1 0 



0 5 12 16 8 0 



8 23 7 3 0 0 



5 9 9 14 4 0 

1 2 0 22 16 0 



12 25 3 1 0 0 



0 5 4 23 9 0 



2 11 6 18 4 0 



3 3 4 22 8 0 



13 25 3 0 0 0 



ERiC 
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Formative Syaluation Questionnaire. (Continued) 



Kev- SA (Strongly Agree), A (Agree), l (Undecided), D (Disagree), 
fjp (strongly Disagreed NA (No Answer). Please circ e your 

choices. sponnd Dav Fifth Dajr 



Second Day 



11. I was stimulated 
to think about 
the topics pre- 
sented. 

12. We worked together 
well as a group. 

13. The group discus- 
sions were excel- 
lent. 

14. There was little 
time for informal 
conversation . 

15. I had no oppor- 
tunity to express 
my ideas. 

16. I really felt a 
part of this 
group. 

17. My time was well 
spent. 

18. The program met 
my expectations. 

19. Too much time was 
devoted to trivial 
matters. 

20. The information 
presented was too 
advanced. 

21. The content was 
not readily ap- 
plicable to much 
research in educa- 
tion. 



SA A ? D SD NA 

10 30 1 1 0 0 

5 13 14 8 2 0 

0 5 17 17 2 1 

3 13 6 16 2 2 

2 4 9 21 5 1 

3 19 13 6 1 0 

6 23 H 1 0 1 

6 17 12 6 0 1 

2 9 8 18 3 2 

0 5 9 21 7 0 

2 9 8 16 4 2 



SA A ? D SD NA 

14 26 0 1 0 0 

5 26 7 2 1 0 

3 15 l 4 8 l 0 

6 6 4 19 6 0 

1 4 4 25 6 0 

3 20 12 4 2 0 

11 22 5 2 0 0 

7 21 3 8 0 0 

3 9 7 16 5 0 

0 5 6 21 7 0 

5 5 20 10 0 
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Formative Evaluation Questionnaire. (Continued) 

Key: SA (Strongly Agree), A (Agree), 7. (Undecided), D (Disagree), 

SD (Strongly Disagree!, NA (No Answer). Please circle your 

choices. 

Second Day Fifth Day 







SA 


A 


9 

# 


D 


SD 


NA 


SA 


A 


9 

• 


D 


SD 


NA 


22. 


The Assistant was 
very helpful. 


9 


18 


13 


0 


0 


2 


14 


18 


7 


0 


1 


0 


23. 


Theory was not re- 
lated t'"' practice. 


0 


11 


Ik 


11 


5 


1 


1 


7 


7 


2k 


2 


0 


2k. 


The schedule should 
have been more 
flexible. 


k 


9 


10 


17 


2 


0 


7 


9 


7 


13 


5 


0 



Ihe "Participant Evaluation Form, " a standard AERA instrument used 
in previous presessions, was administered only once at the close of the 
presession. The tabulation of responses again indicated a general sat- 
isfaction with the presession. The few responses indicating dissatis- 
faction generally refer to one or two specific sessions which could have 

been improved. 

In summary, the results obtained by all of these instruments indi- 
cate a general professional growth on the part of both the participants 
and the staff, as well as a general satisfaction with the presession. 

Director* s Eval uation . This AERA presession "The Computer and. 
Natural Language" is only the second of its kind yet held in educational 
research. In seme ways, therefore, it matched the unusual character of 
its predecessor the year before: its participants came from a variety 

of disciplines; there was a general feeling of adventure and excitement 
and, one mi#it say, a kind of intellectual playfulness in a new study. 
There was, this year as last, a rather remarkable assembly, in which 
there was a wide divergence in professional background and in field of 
application, but often a common focus on natural-language computing. 

That they could all focus on common concerns showed the universality of 
certain computer approaches today. 

There were a n umb er of qualified professionals in the natural-lan- 
guage field, who for same sessions could easily have traded places with 
the instructors, and in some cases actually did so. These participants 
greatly increased the sophistication of some interchanges. But hetero- 
geneity of background is a consideration with almost any such workshop. 

In general, the field had grown up somewhat during the preceding 12 
months of rapid change, and we tried to reflect that growth in the se- 
lection of speakers and topics. More than the former year, we delved 
into computational linguistics, and lingered somewhat longer over the 
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possibilities of artificial intelligence, trying to make explicit the 
eventual connection between AI techniques and inferences in natural- 
language data. And many of the sessions had a somewhat less "experi- 
mental" feeling. Most of the instructors had at various times, in the 
prior year if not before, taught much the same sort of thing in special 
seminars or professional gatherings. 

While the days were often spent in learning particular content, 
the evenings (after the special sessions in enrichment or background 
areas) were often spent in spirited discussion of key problems (such 
as syntactic and semantic approaches, the possibilities of time-sharing, 
etc.). It was an exhausting time, at least for those of us on the full- 
time staff, but the total effect apparently justified the exertion. 

What of the future? The number of applicants approximately doubled 
in one year. Instead of being the most maverick and rarely selected, 
the natural-language session became one of the most widely selected, 
more than filling its quota. This is one more encouraging sign of 
growth in the discipline, and of endorsement of the workshop idea. 

From the staff and participants of this one, some similar sessions 
should emerge, and if possible should be supported. It is an excellent 
topic for such sessions, being new and relatively unknown, having high 
generalizability to a large number of research areas, and being un- 
taught in traditional university programs. 

If such workshops are held, a few comments might be helpful. Last 
year we predicted a drift from FORTRAN to PL/l, but that drift has not 
been as radical as was then anticipated. To the contrary, FORTRAN has 
served just as well as last year for a basic illustrative language — 
perhaps even better, because more programs and conveniences have been 
developed. But in general, languages seemed less important in the in- 
struction; and the concepts of computing, together with concepts from 
various disciplines, seemed much more important. In other words, there 
is same apparent trend from technique to science, making the specific 
languages less important. This trend may be expected to increase, at 
least at the top of the profession. 

A related shift may have made the computer terminals relatively 
unimportant. This year as last, after considerable effort, and consid- 
erable generosity by IBM, the terminals were installed in two locations: 
the pre session library suite, and the lecture area. But this year both 
terminals, to our dismay, went virtually unused. We believe this shift 
represents the increased sophistication of the participants. Another 
year, if we recruited the same kind of advanced students, we would 
probably dispense with the terminals altogether (though we would still 
recommend them for other kinds of groups). 

Participants * Anonymous Evaluations . On the final day of the ses- 
sion 28 participants completed an anonymous evaluation form which was 
administered by and returned directly to the 1969 AERA Research Train- 
ing Sessions Committee. Four items on that form pertained directly to 
the quality of the session. Responses to these questions follow: 

(l) Please rate the quality of instruction in your session. 
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Excellent 36$, Good 54$, Average 10$, Fair 0 $, Poor 0$. (2) Leaving 
aside the quality of instruction for the moment, do you think the topic 
treated in your session should he treated again next year? Yes 93 $> 

No 0$, Uncertain 7$. (3) If you had it to do over again, would you 

apply for the session which you have just completed? Yes 8 9$j No 0$, 
Uncertain 11$. ( 4 ) If a session such as this is held again, would you 

recommend to others like you that they attend? Yes 9 6$, No 0$, Uncer- 
tain 4 $. 
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PRESESSION IV: TRAINING IN RESEARCH ON 

ELABORATIVE METHODS FOR IMPROVING LEARNING PROFICIENCY 



fii - 

OlrUXX 



William D. Rohwer, Jr. University of California 

(Director) Berkeley, California 

Joel R. Levin University of California 

Berkeley, California 

General Description 

The course focused on the knowledge and skills necessary for con- 
ducting experimental research on the problem of improving learning pro- 
ficiency in a variety of populations through methods of verbal and vi- 
sual elaboration. The word elaboration denotes on the one hand those 
learner activities that have been referred to in recent experimental 
literature as associative strategies, mnemonics, natural language media- 
tors, pictorial mediators, syntactical mediators and verbal organizers; 
on the other hand it also denotes properties of stimulus materials con- 
structed so as to bear analogous relationships with the aforementioned 
learner activities. In brief, elaboration refers to the augmentation of 
elements to be learned in the service of more efficient acquisition. 

Given its focus on research training, the course was organized in 
texms of three major kinds of activities: first, the presentation and 

discussion of the current status of research and theory on the problem 
of elaboration; second, demonstrations and practice in the use of 
methods, materials, and procedures that have been employed in such re- 
search; and third, the formulation in research terms of additional ques- 
tions concerning elaboration and the design of experiments likely to 
yield answers to those questions. 

The content of the course, in terms of topics considered, was: 
stimulus properties of learning materials, a variety of task variables, 
subject characteristics as they interact with manipulable elaboration 
variables, and training variables. In connection with all of these top- 
ics, the emphases were on the researchable character of the questions 
that have been asked, the logic relating method to question, the con- 
crete procedures followed to obtain the sought-after answers, and on the 
identity of the issues that still remain to be resolved. Emphasis was 
also given to the implications of research on elaboration for instruc- 
tional tasks and problems such as that of presenting learning materials, 
promoting retention and transfer, and the issue of individual differ- 
ences, in particular, the issue of the culturally disadvantaged. 

Objectives 

✓ 

The objectives of the presession were to encourage research on 
topics related to elaboration and learning efficiency and to promote the 
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acquisition of skills necessary to do so. By way of ac c csnpii shing these 
goals, the aim was to f amil iarize students with relevant research and 
thinking already available, provide examples of foimulating research 
questions and appropriate designs for answering them, provide practice 
in the use of appropriate materials, equipment and techniques, and to 
provide an opportunity to formulate an original research question, an 
attendant design and to select the appropriate materials and techniques 
for conducting the indicated study. Participants were to he able to: 

Objective 1: Accurately report what is presently known and not 

known about elaboration. 



Objective 2: State a research question related to elaboration. 

Objective Describe a design appropriate for answering the 
question. 



Objective 4 : Select appropriate subject populations, tasks, pro 

cedures, materials and equipment for use in the 
stud?/. 



Objective Interpret the results, that is, draw inferences 

frcan them, and relate the results to existing data 
and theory. 



Schedule 

Each day was devoted to two major kinds of activity: two lecture- 

discussion sessions each morning, and a longer, demonstration-practice 
laboratory session each afternoon. The intent was to provide a view of 
the material frcm the vantage point of the experimenter rather than from 
that of an archival reviewer. By using the term lecture -discussion, we 
wish to indicate our intent to stimulate questions and discussion of 
the material presented as it is presented. 



Typically, the afternoon sessions began with demonstrations of 
equipment, materials and procedures by members of the instructional 
staff. Following these demonstrations, students alternately assumed 
the roles of subject and experimenter, conducting four experiments, 
under the supervision of each of the instructors. These miniature ex- 
periments were selected for their relevance to the day’ s lecture-dis- 
cussion and demonstration. 



D^r I 



Session 1: 



Session 2: 



Presession objectives, expectations, evaluation, 
schedule elaboration: definition, history, rele- 

vance, theory, methodology, overview of phenom- 
ena 

Stimulus properties and learning efficiency: I. 

Variations in major modalities, properties of 
pictorial materials, reference to equipment and 
tasks 
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Session 3: Laboratory: Equipment— Demonstrations and pre- 

liminary training in use 



Day II 

Session 1: 

Session 2: 

Session 3: 

Day III 

Session 1: 

Session 2: 
Session 3: 

Day IV 

Session 1: 
Session 2: 
Session 3s 

Day V 

Session 1: 

Session 2: 
Session 3s 



Stimulus Properties and learning efficiency: II. 

Verbal materials 
Single words 
Strings 

Task variables: I* Paradigms (kinds of tasks) 

and instructions 

Laboratory: Samples of all Task Variables 



Task Variables: II. Functions (learning, re 

call), response modes, feedback 
Subject variables: I. Age, Sex, I.Q., E 

Field Trip: PFVT, PA: instructions, action 



Subject Variables: II. SES, Race, E 

Training Studies 

Laboratory: Training techniques and designs 



Sample research proposals with emphasis on train- 
ing studies 

Participant questions and evaluation 
Preparation of proposals and discussion 



Participants 

Seventy-two percent of the participants were male and 18 percent 
female. The average age was 36 years. Those who had previously at- 
tended AERA presessions amounted to approximately l4 percent of the 
group. 

The ma jority of the participants (86 percent) were affiliated with 
colleges or universities. Seven percent were affiliated with the federal 
government and the remaining seven percent with other institutions. 

Acad emi cally, 79 percent possessed the doctorate and Ik percent a 
master* s degree. The group had an average of four articles published 
in scholarly journals and 2.7 funded research projects. 



Materials 

Handouts were prepared for each of the laboratory activities and 
reference materials were available on all of the topics to be covered. 
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Students were asked to read one paper in advance of the presession 
which was distributed by mail. 

The materials included lecture and lab outlines, abstracts of ex- 
periments, reference lists and others such as basic video and sentence 
generation instructions. Exsmp3.es of selected references included 
were: Jensen, A. R. Rote Learning in Retarded Adults and Normal 

Children. -American J * Ment . Defic . , 1965* 69* 828-834; Paivio, A. 
Abstractness, Imagery, and Meaningfulness in Paired-associate Learn- 
ing. J. Verb . Learn « Verb . Behavior, 1965, 4, 32-33; and Rohwer, 

W. D., Jr., and Lynch, S. Retardation, School Strata and Learning 
Proficiency, /mer . J. Ment . Defic . , in press. 

Evaluation 

Tesl Results . The extent to which the objectives were attained 
was evaluated during the last two sessions of the presession which in- 
volved the participants in two activities: preparing a written re- 

search proposal for a single experiment, including a statement of' the 
problem and of its significance, a design, an appropriate methodology, 
and an interpretation of possible out ccmes ; and an evaluation of the 
instruction and the instructors provided by the presession. 

Results cf a questionnaire evaluating the presession are given 
in Table 7» 



Table 7 



Results of Evaluation 



1. Did the presession fulfill your general expectations for it? 

11 - Yes 

4 - Somewhat 
0 - No 

2. In your view what were the two weakest and the two strongest as- 
pects of this presession? (the following responses are the ones 
which appeared most frequently) 

Weakest : 

Lab sessions 

No participant interaction or assessment of participant 
interests 

Lack of organization in library and no way to duplicate 
library materials 
Variability of group 

(Continued) 
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Results of Evaluation (Continued) 



Strongest : 

Organization of materials and presentations 
Excellent references and materials ("handouts") 

Magnificant survey of field 
Laboratory sessions 

3. Were the lecture-discussion sessions sufficiently well-organized and 
coordinated with one another? 

13 - Yes 
1 - Somewhat 
1 - No 

k. Was the content of the lecture-discussion sessions redundant for you? 

1 - Yes 
4 - Somewhat 
10 - No 

5. Was the pace of the lecture-discussion sessions — * 

1 - Too slow 
13 - About right 
1 - Too fast 

6. Did the laboratory sessions provide opportunities for you to acquire 
skills and/or information you wanted? 

9 - Yes 
6 - Some 

7. Were the laboratory sessions adequately coordinated with the lecture- 
discussions? 

8 - Yes 

6 - Moderately 
1 - No 

8. Additional comments, suggestions, criticisms, plaudits: 

A glossary would have been helpful 

Liked informality 

Would have, liked more participant interaction on research 



Director's Eval uation . I am both pleased and dissatisfied with the 
outcome of the" session. Let me begin by listing some of the sources of 
my dissatisfaction: 

1. The number of applicants for the session was excessively small. 

2. My guess is that roughly one-fourth of those who applied where 
somehow misled as to the content of the session. This suggests 
that considerable attention should be given to the phrasing of 
titles and to the possibility of including some descriptive in- 
formation on the publicity sheet. 
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3» The participants were quite heterogeneous with respect to rel- 
evant preparation. This heterogeneity had only one really 
negative consequence for the session— the discussion was 
usually dominated by those having more sophistication, some- 
times to the exclusion of the other participants. 

k. With more foresight, we could have facilitated the acquisition 
of a common temninology among the participants by providing a 
glossary. 

5. More time should have been devoted to joint staff -participant 
consideration of participant research proposals. The cost of 
doing this would have been the deletion of some information 
about previous research!. Such a price would have been wel±- 
justified in view of the fact that the ability to formulate, 
evaluate and refine research ideas was the major objective of 
the session. 

6. Finally, a relatively minor point — facilities for duplicating 
reference materials would have enhanced the session. 

In contrast to these matters, I count the following aspects of the 
session as quite gratifying in view of our purposes and the goals of 
the participants: 

l. In accomplishing the task of making the wide variety of ar- 
rangements necessary for our session, the assistance of the 
AERA staff support person was indispensable. 

2. Although economically unsound, the small size of the session 
was a boon for the kind of participant discussion and staff- 
participant interaction we wished to foster. Frequently the 
discussions that arose in connection with the lecture presen- 
tations were unusually st imu lating. 

3. The heterogeneity of the participants had one major salutary 
effect, namely, that the teaching functions were spread among 
the participants and not confined to the staff. Furthermore, 
the varied kinds of expertise represented among the partici- 
pants augmented the amount of relevant information available 
in the session over what could have been provided by the staff 
alone. 

4. Although the evaluation sheets indicated that the laboratory 

sessions were a mixed success, their role in the session was 
a very important one: they provided an opportunity for the 

less sophisticated participants to acquire information and 
skills already possessed by the more sophisticated ones. 

Thus, to a considerable degree, the use of two contrasting 
kinds of activities, lecture-discussion and laboratory, pro- 
vided a solution for same of the problems attendant upon the 
varied initial skills of the participants*. 
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5. The laboratory session in which young children were brought 
in to serve as subjects in demonstration experiments was a 
particularly successful one<> 

6. The discussion of participant research, proposals during the 
final session were quite fruitful. The group was sm all enough 
so that each proposal was heard by and discussed with at least 
two members of the staff and with other interested partici- 
pants as well. The proposals themselves were of good quality. 

7. Because of our small enrollment, it was possible to accommo- 
date a number of graduate students. My impression is that 
they gained as much or more from the session as did the post- 
doctoral participants. 

Participants’ An onymous Evaluations . On the final day of the ses- 
sion l£ participants completed an anonymous evaluation foim which was 
administered by and returned directly to the 1969 AERA Research gain- 
ing Sessions Committee. Four items on that fom pertained directly to 

quality of the session. Responses to these questions follow: 

(1) Please rate the quality of i .'.struction in your session. Excellent 
36$ Good 43$, Average 1 4$, Fair 7$, Poor 0 $. (2) Leaving aside the 

quality of instruction for the moment, do you think the to^ic treated 
in your session should be treated again? Yes 50 $, No l4%, Uncertain 
3 g* (3) if you had it to do over again, would you apply for the ses- 
sion which you have just completed? Yes 71$, No 22$, Uncertain 7%« 

(4) If a session such as this is held again, would you recommend to 
others like you that they attend? Yes 71$, No 22$, Uncertain 7%. 




PRESESSION V: SYSTEMS APPROACH IN 

COUNSELING AND COUNSELOR EDUCATION 



Staff 



T. Antoinette Ryan 
(Director) 

Harry Anderson 



University of Hawaii 
Honolulu, Hawaii 

University of Georgia 
Athens, Georgia 



Piay E. Hosford 



University of Wisconsin 
Madison, Wisconsin 



Leonard C. Silvern 



Education and Training Con 
sultants, Co. 



Carl E. Thoresen 



Stanford University 
Stanford, California 



General Description 

This pression in systems research vias offered in response to re- 
quests from counseling personnel and counselor educators. It was de- 
signed as an advanced program focusing on the use of systems research 
in planning and evaluating counseling and counselor education programs. 
The program was planned to equip counseling personnel and counselor 
educators with practical skills and theoretical knowledge essential 
for implementing systems research at local district, state department 
or university levels. The course of study dealt with conceptualiza- 
tion of systems research, application of systems research, techniques 
of systems research, and practical uses of systems research. 



Objectives 

The purpose of the presession on systems approach was to achieve 
improvement and innovation in counseling and counselor education 
through research. The long-range outcome expected as a result of the 
training session was initiation of positive change in counseling and 
counselor education through systems techniques. The overall purpose 
was implemented via four immediate objectives. Participants would: 

Objective 1: Become familiar with concepts and principles of 

systems research, as revealed by meaningful use of 
concepts including analysis, synthesis, logistics, 
simulation, feedback, terminal behaviors, and 
modelling. 

Objective 2: Gain knowledge of applications of systems research, 

as revealed by designing a closed-loop system with 
feedback between and within major subsystems, relat- 
ing parts to wholes, and designing graphic analogs. 
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Objective Develop skills in applying techniques of systems re- 
search, including flow chart symbols (signal path 
connections, signal path directions, arrowhead for- 
mation, arrowhead connections, rectangular blocks, 
descriptor length, coding by point-numeric, upper 
case lettering, F, FF, A, error signals), and iden- 
tification of significant elements. 

Objective 4: Develop favorable attitudes toward systems research, 

as revealed by reactions to concept terms. 

Schedule 



2 SLl 



Session Is General Session 1 

Overview, Pretest, Systems research in counseling 
and counselor education 



Session 2: Topic 1 - Systems Approach : An Overview 

Conceptualization of system in model form. Gen- 
eral model of instructional system, operational 
definitions 

Session 3: Model for producing a system model. Systems using 

feedback 



Session 4: Topic 2 - Appl ication of Systems Research 

Non-school setting. School setting: Quantitative 

applications 

General Session 2 

Introduction of research persons. Task assign- 
ment 



Day II 

Session 1: 

Session 2: 
Session 3i 
Session 4: 



Topic 3 - Tech niques of Systems Research 
Defining Behavioral Objectives, Conceptual 
Analysis and Synthesis Analysis, Synthesis 

Task assignment. Individual activity: Analysis, 

Reports, Task groups 

Operationalizing Strategies, Rules and symbols 
for flowchart modeling. Task assignment 

Task assignment : Development of flowchart 

model. Reports 



Day III 

Session 1: Topic 4 - Case St udies in Systems Research 

Case Study: General Problems, Analysis and 

synthesis of problem 
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Session 2: 



Task group reports* Case Study: Counselor Educa' 

tion System* Modeling and simulating to test 
model 

Session 3: Task group reports 

Session 4: Case Study: Counseling and Guidance, Modeling 

and simulating to test model and solve problems. 
Informal meeting 



Day IV 



Session 1: 
Session 2: 
Session 3: 



Session 4: 



Task group reports 
Critique of solutions 

Review of concepts and principles* Systematic 

counseling _ , 

Topic 5 ~ Application of System Research to^ 

lems Identified by Participants 

Task group work on problems. Staff and partici- 
pant meeting 



Day V 



Session 1: Task force group reports 

Session 2: Program evaluation 

Session 3: Systems evaluation* Summary and conclusions: A 

review and preview 



Participants 

Thirty-six applicants were accepted for the presession, including 
34 regular participants and two alternates. Two of the 34 accepted 
for regular participation were unable to attend; thus, two alterna es 
were invited to participate, making a total enrollment of 34 trainees. 



The participants ranged in age from 22 to % years, with a median 
aee of 37 years* and included 33 males and one female# oixty- wo per 
cent of the participants held the doctorate, while 12 percent were 
doctoral candidates. The remaining 2 6 percent held a master s degree. 
The majority (62 percent) were from colleges or universities. _ Local 
school districts were represented by 26 percent of the participants, 
while three percent were from state department of education, and nine 
percent from other fields. 



Materials 

Advance materials were sent to participants prior to the opening 
of the training program, to assist participants in acquiring a common 
background of knowledge relevant to the topic. With the a ® ce P^ a “° e 
notice each participant received a syllabus and bibliography ong 
other forms. Participants were asked to read the text. Systems 
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Engineering of Education Is Evolution of Systems Thinking in Educa- 
tion, 11 by L. C. Silvern, before the opening meeting. 



The rec omme nded reading included such references as Preparing 
Instructional Objectives by R. F. Mager, The Systems Approach and 
Counselor Education; Basic Features and Implications by C. E. 
Thoresen, and M Look at Education : System Analyses in Our Schools 

and Colleges by J. Pfeiffer. Other instructional materials included 
films, slide -tape presentations, and programmed materials. 

Evaluation 



Test Results . Two criterion measures were taken at the comple- 
tion of the five-day training session to determine extent to which 
immediate objectives had been achieved. A pre- and posttest con- 
sisting of six sub-tests was administered to assess participant knowl- 
edge of concepts and principles, understanding of application of prin- 
ciples, and skill in using systems techniques. A post-training inven- 
tory was administered to determine degree to which feelings about sys- 
tems concepts were positive. Pretest results identified participants* 
level of pretraining knowledge of systems research, and served as 
guide to adjust instructional program in terms of learner characteris- 
tics and needs. 

Posttest results of the Systems Evaluation sub-tests were ana- 
lyzed to determine extent to which participants developed knowledge 
and skills of systems research. The relation between sub-tests and 
training objectives is shown in Table 8. 



Table 8 

Relation Between Systems Evaluation Sub-tests and Training Objectives 





Sub-test 




Training Objective Measured 


No. 


Title 


No. 


. Description 


1 


Use of Flowchart symbols 


3 


Use of systems techniques 


2 


Application of Principles 


2 


Application of principles 


3 


Understanding Concepts 


1 


Understanding concepts 


4 


System Complexity 


3 


Use of systems techniques 


5 


Element Identification 


3 


Use of systems techniques 


6 


System fidelity 


3 


Use of systems techniques 



Acceptable performance criteria were defined for objectives re- 
lating to knowledge and skills of systems approach. Sub-test scores 
constituted acceptable performance criteria for knowledge and skills 
objectives and percent of participants meeting criterion levels on 
the posttest are reported in Table 9. 



Table 2 



Performance Criteria for Three Training Objectives and 
Percent of Participants Meeting Posttest Criterion Levels 



Objective Posttest Criterion Levels of Acceptab le Performance 



No. Description 


Posttest 

Number 


Possible 

Score 


Criterion 

Level 


$ Achieving 
Criterion 
Performance 


1 Understanding 
concepts 


(3) 


15 


12 


47 


2 Application of 
Principles 


(2) 


20 


16 


100 


3 Use of tech- 
niques 

Total 


(1) + (5) 


35 


28 


82 


70 


56 


82 



Sub-tests 4 and 6 of Systems Evaluation Test measured skill in^ 
dealing with system complexity, and skill in achieving system fidelity, 
respectively. Analysis of results of sub-tests 4 and 6 reveal that out 
of a possible twenty points combined score for the two sub-tests, forty- 
seven percent of participants reached 80 percent criterion level of 16 

or higher. 

The System Evaluation instrument was pilot tested on six subjects 
who had previously completed a five-day training program in designing 
education and training systems. Mean scores on the six sub-tests ob- 
tained on the pilot testing of the instrument and mean scores on post- 
tests of participants in the AERA presession compare as follows: 



Sub-test 


Pilot 


Test 


AERA 


Presession 




X 


Possible 


X 


Possible 


1 


16.00 


20 


18.47 


20 


2 


17*16 


20 


19*58 


20 


3 


11*33 


15 


11*13 


15 


J 

4 


7*83 


10 


4.64 


10 


5 


12.16 


15 


10.64 


15 


6 


1?*82 


20 


10.00 


10 


Total 


80.33 


100 


74.47 


90 



The total possible score for the pilot test differs by ten points from 
the total possible score for the presession posttest, because of a ten 
point difference in scoring of Sub-test 6* When scores for Sub-test 6 



ERIC 
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are eliminated from the totals, the mean total score for the pilot 
test of 64.50 compares to a mean total score on the presession post- 
test of 64.^7. 

Three self-evaluation items were included on the program Inven- 
tory, to see how participants felt about the progress they made to- 
ward tra inin g objectives. Table 10 reports the percent of partici- 
pants who felt gains in knowledge or increase in skill proficiency re 
suited from the program. 



Table 10 

Percent of Participants Who Felt They Reached Four 
Performance Levels of Training Objectives 





Attainment Level 




Objective Description None 


Very 

Little 


Quite 
A Lot 


Great 

Amount 


No 

Response 


I. How much new knowledge 0 


6 


58 


30 


6 



about systems research 
do you feel you ac- 
quired during this 
program. 



2. To what extent do you 6 

feel knowledge you had 
before presession was 
recognized or reinte- 
grated. 


14 


21 


53 


6 


3. To what extent do you 0 

feel you developed 
proficiency in using 
systems techniques. 


12 


73 


9 


6 



Results in Table 10 indicate that 88 percent of participants felt 
the program resulted in their acquisition of new knowledge, and 81 
percent felt the program led to proficiency in using systems techni- 
ques. Results reported on self-evaluations are supported by compari- 
son of pre- and posttest scores on the criterion test. Out of a total 
possible score of 90 , mean pretest score on the Systems Evaluation 
Test was 15.12, with variance of 242.28, compared to a posttest mean 
of 74.47, and variance of 22.14. 

Results indicate immediate training objectives concerned with 
developing participant knowledge and skills in systems research were 
attained at criterion levels by 82 percent of participants. 
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Analysis of post-training results on the Systems Evaluation Test 
revealed that 47 percent of participants reached an 80 percent criterion 
performance level for acquisition of knowledge about systems research 
concepts. 

Results of Systems Evaluation Test indicate that 100 percent of the 
participants reached the 80 percent criterion level on test of applica- 
tion of principles of systems research. 

Analysis of results revealed that 82 percent of the participants 
reached the 80 percent criterion level on tests of skill in using sys- 
tems techniques. 

An extensive program evaluation was also made to deteimine the ex- 
tent to which various program components contributed to effectiveness 
of the pre session. Data were gathered to evaluate learning activities, 
instructional materials, lecture content, and program organization. 

Directors Evaluation. Results from the post-training criterion 
tests a dmin istered immediately following conclusion of the program in- 
dicate that achievement of three of four training objectives reached 
criterion level. Criteria of acceptable performance were defined for 
training objectives relating to development of participant knowledge and 
skills. Eighty-two percent of participants reached criterion levels of 
performance. Analysis of results revealed that understanding of systems 
concepts was at a lower level than application of principles or use of 
techniques. Analysis of responses to the sub-test on Understanding Con- 
cepts revealed a general lack of understanding of synthesis and logistics. 
This problem might have been overcome had more use been made of dally 
evaluation to identify participant gaps in knowledge in sufficient time 
to devise learning experiences aimed at bolstering specific weaknesses 
in concept understanding. This problem also might be tackled through a 
more rigorous schedule of required reading. Only one reference was re- 
quired. In participant evaluations of recommended readings no responses 
were given by over two-thirds of participants for four of five recom- 
mended references, suggesting that participants had not implemented the 
recommendation to study these materials. 

The finding that 82 percent of participants reached criterion level 
of performance in use of systems techniques suggests viability of learn- 
ing activities designed especially to develop skills in systems research. 
The task group assignments, in which participants worked as task groups 
with members of the presession staff assigned to implement instructional 
role for each group, were designed to provide supervised practice in use 
of syst ems techniques. Results from criterion tests suggest this ap- 
proach has merit for developing skill proficiency. Results from the two 
sub-tests d ‘signed to measure skill in dealing with system complexity and 

in achieving system fidelity suggest that participants failed to reach 
a particularly high level of sophistication in these areas. It is hy- 
pothesized that performance level is a function of time, and that per- 
formance level is a function of time, and that either more attention to 
pre-program reading or more t ime allotted to the presession proper might 
lead to more sophisticated performance on the part of trainees. 
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It was not expected that a particularly significant impact would 
he made on fostering of attitudes favorable to systems research, due 
to the limitations of time. It would he expected that as participants 
implement systems techniques in connection with their professional 
role responsibilities, attitudes toward systems concepts would become 
more favorable. 

There was same indication of dissatisfaction with the physical 
arrangement of the room. However, no comments were given to indicate 
the focus of this response. The general classroom climate appeared 
to be conducive for learning, with participants reacting favorably to 
the extent to which freedom for expression was allowed and opportunity 
for participants to make new acquaintances was fostered. Interest in 
the AERA sponsored pre sessions was strong, with 91 percent expressing 
the opinion that a presession on systems approach should be offered 
again, find eighty percent indicating interest in attending a preses- 
sion next year. 

There are indications of ways in which the training program might 
be improve^ However, there is strong support both in results of 
tests of participant performance related to achievement of immediate 
training objectives and participant evaluation of program organiza- 
tion and management that the program generally was viable and that 
attainment of long-range goals, the improvement of counseling and 
counselor education through research, should be obtained. 

Participants 1 Anonymous Evaluations . On the final day of the 
session 30 participants completed an anonymous evaluation form which 
was administered by and returned directly to the 19&9 AERA Research 
Training Sessions Committee. Four items on that form pertained di- 
rectly to the quality of the session. Responses to these questions 
follow: (l) Please rank the quality of instruction in your session. 

Excellent 6jf>, Good 27%, Average 7%, Fair 3% Poor 0%. (2) Leaving 

aside the quality of instruction for the moment, do you think the 
topic treated in your session should be treated again next year? Yes 
9^%> Ho 3%, Uncertain 3 %• (3) If you had it to do over again, would 

you apply for the session which you have just completed? Yes 90%, 

No jf 0 , Uncertain 6%. (4) If a session such as this is held again, 

would you recommend to others like you that they attend? Yes 97%, 

No 0%, Uncertain 3$» 
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PRESESSION VI: MULTIV ARIATE RESIGN 

AMD ANALYSIS IN EDUCATIONAL RESEARCH 



Staff 



Joe H. Ward, Jr 
(Director) 



Southwest Educational 
Development Laboratory 



Robert Bottenberg 



Air Force Personnel Research Lab 
Lackland Air Force Base, Texas 



Clyde P. Donahoe 



Systems Development Corporation 
Santa Monica, California 



Earl Jennings 



Computer Assisted Instruction Lab 
University of Texas 
Austin, Texas 



General Description 



This session was designed to develop an appreciation for multiple 
linear regression as a general approach to the formulation and analy- 
sis of research problems. As such, the activities were divided about 
evenly between lecture-discussion, laboratory exercises related to the 
objectives listed, and exercises related to appropriate computer opera- 
tions. The illustrative problems were drawn largely from the behav- 
ioral sciences. 

Concepts and exercises were introduced systematically as they were 
required in the logical development of the materials. Participants had 
direct experience with data processing and computer equipment. Each 
participant prepared a problem statement which reflects acquisition of 
concepts and development of the attendant techniques that are useful 
in conceptualizing research problems. 

There are two major reasons for the lack of use of general regres- 
sion models. First, there have been few attempts by teachers to de- 
velop the behaviors in students that are necessary to effectively create 
models appropriate to the particular problem of interest. Second, many 
of the models that should be utilized for a particular problem require 
the use of a computer, but many research workers do not have effective 
systems for communicating with the computer. These two problems can be 
alleviated by (l) providing an instructional system that will develop 
in students the capability of defining regression models appropriate to 
their problems of interest; and (2) providing computational soft-ware 
that facilitates the analyses by high-speed computer. This course was 
designed to assist in both problems. 



Objectives 



The -pr imar y objective of this session was to assist the partici- 
pants in developing techniques of formulating research problems for 
computer analyses so as to make full use of the multiple linear re- 
gression approach. Participants were to be able to: 

Objective 1: Define vectors that express his conceptualization 

~ of a problem. 

Objective 2: Formulate models appropriate for his specific prob- 

lems without conforming to experimental designs 
for which prescribed computational procedures are 
available. 



Objective 3: Identify vectors that represent information that 

has been measured on a continuum. 

Objective 4: Define vectors so as to express nonlinear and in- 
teraction relationships. »- 



Objective 5: Use categorical and continuous vectors^ in models 

developed to remove the n contamination ,f of other 
factors (logic of convariance analysis). 



Objective 6: 



Apply an unambiguous set of rules to the determina- 
tion of the appropriate degrees of freedom to be 
used with the linear regression model. 



Objective 7: Cite novel examples of research problems to which 

linear regression is applicable. 



Schedule 






Session 1: 



Session 2: 

Session 3* 
Session 4: 

Session 5* 
Session 6: 



Background and objectives* Research Analysis 
Lecture (RAL), Chapter 1 of”"Applied Multiple 
Linear Regression" 

RAL* Chapter 2 through Section 2.4, Mutually 
Exclusive Categorical Models 

RAL, Linear Dependence: definition and examples 

RAL, Comparison of Assumed and Restricted Models, 
Computation of the F-Statistic 
Computer Analysis Lecture (CAL) 

CAL, Discussion of three Services Card Deck and 
output. Input three Services Problem with DATRAN, 
Problem Set 1, Problem Set 2 



Day II 

Session 1: 



RAL - Problems in Ordering and Linearity based 
on Mutually Exclusive Categorical Models 
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Session 2: 

Session 3 • 
Session 4: 
Session 5: 



CAL - Preparation of Computer Input for Problems 

in Ordering and Linearity, Input computer deck 

Summary of morning. Discuss output 

RAL - Assumptions of General Linear Model 

HAL - Assumptions of General Linear Model, Problem 

Set 3 



Day III 



Session Is RAL - Extensions of 1 - Attribute Problem 
Session 2: RAL - 2 - Attribute Problems 

Session 3: LAB. Individual work - 2-way analysis - Prepare 

own models, f -Cards, etc. 

Session 4: Further variations on the 2-way analysis 



Day IV 



Session Is 
Session 2: 
Session 3: 
Session 4: 
Session 5* 



RAL Covariance Analysis 
Covariance Analysis 
3 - Factor Interaction 

RAL - Continue on Covariance Analysis (variations) 
RAL - 2 Attribute Problem - Linearity assume in 
both attributes 



Day V 



Session 1: 

Session 2: 
Session 3 * 
Session 4: 



RAL - F-Statistic, Orthogonal Decomposition and 
Least Squares Computation 

RAL - Additional Application of Regression Models 
Summary and Evaluation 

Demonstration of Application of SDC time-sharing 
computers to Educational Problems. Computer as- 
sisted instruction, instructional management, 
financial management, and other applications. 



Participants 

Eighty-six percent of the 58 participants were male, and l4 per- 
cent female. They had an average age of 36 . 8 . Approximately 67 per- 
cent held a doctorate. Their research productivity was reflected in an 
average of 3.3 articles published in a scholarly Journal. They also 
had an average of .9 funded research project per participant. 

Geographically, the majority of participants (44 percent) came from 
the Midwest 5 fifteen percent came from the South 5 20 percent from the 
East Coast; two percent from the West Coast; five percent frcm the South- 
west and five percent from the Northwest. Canada contributed seven per- 
cent and Australia two percent. 
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Materials 



Examples of materials distributed were: DATRAN Explanation Sheet, 

How Chart of a Program, Assumptions Underlying Fixed X Model, Control 
Cards for Quadratical Covariance, and Personnel Research Lab Publica- 
tions. 

Evaluation 



Test Results. A result of previous evaluation efforts of the pre- 
session was the mailing of the Bottenberg and Ward text. Multiple Lin- 
ear Regression, before the session started. One difficulty with the 
pre-mailing was that a text couldn’t be mailed until a participant was 
selected; if an application was made late, the text would then con- 
sequently be mailed late. However, a pretest showed the following re- 
sults : When asked, "Did you read it?, " eight wrote that they did, seven 
wrote that they did not, nine wrote that they sk imme d it, and 32 failed 
to nake any response. The pretest involved defining three vectors, 
filling in the appropriate binary coded digits, and finding the error 
vector. 

After a rather liberal grading by one of the staff members, the 
following distribution of scores was found: 

All three correct: 18 

two correct: 12 

one correct : 15 

none correct : 11 

A mean of 1.66 was found. The pre-mailing was somewhat less than 
a howling success. Considered from another vantage point, however, the 
pre-mailing was useful. While a participant may not have read the text, 
he was at least aware of the direction that the presession might take. 

Immediately after the presession had been completed, the partici- 
pants filled out an evaluation form. Pertinent questions are repeated 
in Table 11 with totals regarding each question. 



Table 11 

Participants’ Evaluation of Content and Presentation 



1. Did the content of the lectures and readings presuppose far more 
previous training than you had? 

Yes 8, No 34, Occasionally l4 

Should less training in these areas or more have been presupposed? 
Less 8, More 5, Same 29 
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Table 11 (Continued) 

2. To what extent was the content of the lectures and readings rele 
vant to what you hoped to accomplish during the session? 

Favorable response 45, Unfavorable response 0, Neutral 7 



3* Were the lectures stimulating and interesting? 

Yes 46, No 0, Usually 7 

Were the lecturers competent to speak on the subject assigned them? 

Yes 55, No 0, Usually 1 
Were the lecturers well prepared? 

Yes 53, No 0, Usually 3 

4. Were you disappointed in any way with the group of participants? 

Yes 3, No 50 

5 . If you had it to do over again would you apply for this pre session 
which you have just completed? 

Yes 54, No 1, Unsure 1 

6. If a presession such as this is held again would you recamend to 
others like you that they attend? 

Yes 54, No 1, Unsure 1 

7. Do you anticipate maintaining some sort of contact with at least 
one of the presession staff? 

Yes 46, No 8, Unsure 2 



8 . 



9 . 



Do you feel that AERA is making an important contribution to edu- 
cation by sponsoring presessions such as this one? 

Yes 56, No 0 

Do you feel that anything has happened during these five days to 
make it more likely that you will leave your present position of 

employment? 

Yes 11, No 44, Unsure 1 



10. Is it likely that you will collaborate in research with someone 
else attending this presession (other than those you already were 
likely to collaborate with)? 

Yes 20, No 31* Unsure 5 



Do you feel that the staff should feel that it has accomplished 
its objectives during this five-day presession? 



Yes 54, No 1, Somewhat 1 



11 . 



r 



Director’s Evaluation . The effectiveness of a presession is not 
necessarily easily measured 5 the question of the criterion is elusive. 
What is a presession supposed to do? This can be stated in terms of 
specific objectives to be accomplished. However, specific objectives 
over a defined time period may not be as important as other effects, 
perhaps unknown at the time of a presession. Also a presession can 
accomplish in rather fine fashion the stated objectives; however, if 
the staff has so much closure that it is no longer viable, little else 
but those objectives will ever result. That this staff itself was 
viable is evidenced by their decision to pre-mail the text. Other sug- 
gestions made by participants seem also to have been given considera- 
tion. If the advanced session planned for 1970 becomes a reality, the 
staff will have again shown their ability to accept and act on any 
suggestions. 

It is just as important, if not more important, for the partici- 
pants th ems elves to leave the presession without having became closed 
minded about their approach to research. If the material is as new 
tc the participant as would seem to be indicated by their comments, 
considerable time will be needed to put the ideas presented in proper 
perspective. Hopefully, this will contribute to their competencies 
and practices in educational research. 

The chairman of the Research Training Sessions C omm ittee provided 
overall coordination that facilitated the conduct of the pre session. 

The method of processing the applications worked satisfactorily. It 
was decided during the presession director’s meeting in Chicago that 
there would be no deadline for applications. X think that it is de- 
sirable to have a deadline for the applications . This deadline could 
be accompanied by a statement that "applications received by the dead- 
line will be given first consideration, however thOi^e rec i^f ( ^ 

the deadline w ill be considered if space is available . ” If the appli- 
cants have a deadline they could get firm c o mmitment for their atten- 
dance. A deadline would help the presession directors who wish to send 
materials for advanced study. 

There should be consideration given to the values derived from 
the director’s meeting in Chicago in the fall. It might be interesting 
to try a co nf erence phone call. It is nice to get to meet directors 
for the first t ime ; however, there should be some way of determining 
the expected payoff of the gathering. Perhaps there should be a meet- 
ing of all new presession directors plus one experienced presession 
director. I suggest some type of cost-benefit consideration concern- 
ing the director’s meeting in the fall. 

Even though the following comment was made in previous years, it 
needs repeating. Even though the entire staff enjoys putting on the 
presession, it is apparent that much more staf f time is required in 
pre session planning than is allowed for in the budget . It is fortunate 
that the employers of the pre session staff are willing to provide the 
staff with the time required to carry out the necessary planning. 
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The 1969 presession planning not only eliminated the need for most 
of the key punching required by participants in the 1967 presessian, but 
also e limin ated the need for. same of the computer runs that were re- 
quired in the 1968 presession. Approximately one-half of the computer 
runs were simulated. All of the output that would result from the runs 
were prepared in advance so that instruction would not have to depend 
entirely upon computer scheduling. This simulation was also stimulated 
by the inadequate operating system of the IBM 360 system at System De- 
velopment Corporation. It would have cost entirely too much money to 
have act ually executed all of the programs of the presession. It turned 
out that it was most economical to go outside System Development Cor- 
poration for computer service, since the required service could be ob- 
tained at much less cost. As a result of this partial s imul ation, it 
is pl anne d that future presessions will have more than half of the runs 
simulated. In very short sessions possibly one-hundred percent of the 
computer runs can be simulated. While it is certainly desirable to have 
seme of the probl ems actually run during the educational program, it 
se ems that a five-day session is a difficult time span to require com- 
puter activity. A session that lasts from one to two weeks would cer- 
tainly allow sufficient time for running problems on the computer. 

The presession can be called "successful" as a result of analysis 
of the comments supplied at the end of the five days. Most of the com- 
ments were quite favorable and many participants had useful suggestions 
for possible improvement. 

The use of transparencies was mentioned as a help for presentations 
during the presession. It is planned to exper^rient with a mixture of 
student listening and writing in future preses,- ions. This year much 
writing was done by the participants; however, it seems reasonable that 
seme writing alternated with sane listening and thinking would be more 
beneficial. 

It seems that our effort to send the major publication in advance 
of the presession was well received and I am sure we will send advanced 
materials in future years. This experiment came as a result of previous 
evaluation comments in which many participants suggested advanced mail- 
ing of reading materials. 

In 1968, one of the participants returned to his own university to 
introduce this approach to problem analysis immediately. Within two 
weeks of the presession he had received same positive feedback from his 
own strident s. A second faculty member from the same institution at- 
tended the 1969 presession. As a result, plans were made the week fol- 
lowing this presession to have a staff member present a two-day course 
of instruction at the university. 

There seems to be much interest at the present time in a second 
type of instructional program — one for those people who have finished 
a first pre session. There are approximately 160 people who have at- 
tended one of the three previous presessions (or its equivalent). Many 
of these participants would like an intensive study program that would 
allow them to apply to other problems what they now know. 
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Participants * Anonymous Evaluations * On the final day of the ses- 
sion 55 participants completed an anonymous evaluation form which was 
administered by and returned directly to the 1969 AERA Research Train- 
ing Sessions Committee . Four items on that form pertained directly to 
the quality of the session. Responses to these questions follow: (l) 

Please rate the quality of instruction in your session. Excellent 62 $, 
Good 31$, Average *J$, Fair 4$, Poor 0$. (2) Leaving aside the quality 

of instruction for the moment, do you think the topic treated in your 
session should be treated again next year? Yes 99$, No 0$, Uncertain 
2$. (3) If you had it to do over again, would you apply for the ses- 

sion which you have just completed? Yes 88$, No 2$, Uncertain 10$. 

(4) If a session such as this is held again, would you recommend to 
others like you that they attend? Yes 94$, No 2$, Uncertain 4$. 



PRESESSION VII : A NTHROPOLOGICAL 

FIELD METHODS IN EDUCATION RESEARCH 



Staff 



Harry F. Wolcott 
(Director) 



Center for the Advanced Study 
of Educational Administration 
Eugene, Oregon 



Alfred G. Smith 
Louis M. Smith 



University of Oregon 
Eugene, Oregon 

Central Midwestern Regional 
Educational Laboratory 



George D. Spindler 
Louise Spindler 



Stanford University 
Stanford, California 

Stanford University 
Stanford, California 



General Description 

Increasing attention has been shown in recent years in applying 
the field methods of the cultural anthropologist as a perspective for 
inquiry into formal educational settings. The purposes of this pre- 
session were to investigate the field methods which anthropologists 
use, to consider the possibilities and limitations of several specific 
methods and approaches, to be able to engage in a dialogue with people 
who have used or who would like to use these techniques, and to consider 
the suitability of such techniques for the specific research interests 
and competencies of the participants. Content was concerned with such 
questions as the following: What anthropological field methods are 

appropriate in educational research? What are the procedures and prob- 
lems with the specific methods identified? What kind of results have 
been obtained from such methods? Who is doing or planning such research 
at present (an information exchange)? How does one "get started" in 
using this approach? 



Objectives 

The major objective of the session was to acquaint participants 
with a variety of field techniques used by cultural anthropologists, 
drawing upon specific examples of fieldwork in. which the contributing 
staff members had engaged. At the close of the session these objectives 
should have been attained: 

Objective 1: To grapple constantly with the problem in fieldwork: 

~ What do you look for? 



o 

ERIC 
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Objective 2: 



To provide adequate time and opportunities for par- 
ticipants to discuss their specific problems with 
staff and with each other. 



Obi active 3: To assess the relevance and limitations of any field 

study methods introduced, and to identify what con- 
stitutes a distinctly anthropological study from a 
non-anthropoiogical one. 



Schedule 



Day I 

Session Is Planning Conference (Staff Only) 

Session 2: Introduction of staff. Introduction of partici- 
pants: research interests, methodological pro 

lems. Staff round-table and orientation for^the 
session, critique of "Sane Conceptual Tools" by 
A. G. Smith 



Day II 



Session 1: 



Session 2: 



Micro-ethnography of the classroom: (l) Com- 

plexitie s of an Urban Classroom, (2) Teacher 
plans and classroom interaction. Panel discus- 
sion— A Kwakiutl Village and School, Open dis- 
cussion • 

Field Studies in psyco-cultural adaptation and 

cultural transmission, (l) Techniques: Case 

st udi es and participant observation, (2) Sources: 
Fieldwork with American Indians in a German Vil- 
lage and in West Coast schools. Panel discussion 



Day III 

Session 1: The case as method in anthropology. Panel dis- 

cussion— Case Studies in Cultural Anthropology , 
Case Studies in Education and Culture, Open dis- 
cussion 

Session 2: Field studies in psycho- cultural adaptation ana 

cultural transmission, (l) Techniques: Projec- 

tive techniques, Rorschach, Instrumental Activi- 
ties Inventory, Expressive autobiographic inter- 
views 



Day IV 

Session 1: Use of video-tapes in training observers. Demon- 

stration and discussion 

Session 2: Study of a computer assisted instruction program 

Appalachia, Panel discussion 



Day V 



Symposium: Anthropological studies of the organization and 

administration of education: Kensington, an 

innovative elementary school; The ethnography 
of a principalship; Studies of West Coast Schools, 
Analysis through communication networks: Ccm- 

munication and Status 



Participants 

Averages from 92 percent of the 31 participants revealed the fol- 
lowing information. Seventy-one percent were male and 29 percent fe- 
male. The average age was 36.4 years. Approximately 29 percent of the 
participants had previously attended AERA pre sessions. 

Seventy-one percent were from a college or university. The public 
school systems contributed 20 percent of the participants, the federal 
government two percent and state departments of education two percent. 

The majority of the participants (74 percent) held the doctorate, 
while the remainder held masters degrees. The average number of arti- 
cles published in a scholarly journal was 2.7 and the average number 
of funded projects per participant was .9* 

Materials 



Instructional and evaluation materials used were: a discussion 

stat eme nt prepared to open the session, bibliographies and discussion 
materials prepared by staff members for use in specific sessions (e*g. ; 
Alfred Smith: Bibliography: "A Few Diverse Guides to People Watch- 

ing” ; programmed sequence of questions, "The Case as Method in Anthro- 
pology"), a monograph distributed to all participants, and audio-visual 
materials. There were also exhibit copies of several books and mono- 
graphs from three series: Case Studies in Education and Culture, Case 

Studies in Cultural Anthropology, Studies in Anthropological Method . 

Evaluation 



Test Results * A "Participant Reaction" form was completed by 28 
participants on the last day of the session. Representative responses 
were as follows: 

1. Did the content of this session have direct and immediate ap- 
plication to your own work in the research and practice of 
education? 

a. Enthusiastic : "Yes, I*m particularly interested currently 

in the influence of the sub-culture of the school on the 
new teacher and the session has suggested a number of leads 
regarding appropriate questions to ask and methodologies to 
apply. " 
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b. 



Unenthusiastic: "Yes, but very little, far less than I 

had expected— wished! It tended to he ’old stuff* or 
’Mickey Mouse’— very poorly presented in many instances." 



Can you anticipate any long-range consequences or outcomes 
for you as a result of your participation in this session? 



a. Enthusiastic : "I’ve been interested in the anthropological 

analysis of education for some years and the session has 
brought the interest to the fore. I’ll be catching up 
with the literature in the field in the immediate future 
and would like to test some of the ideas I’ve gleaned in 
sane research I’m currently planning. Also planning to 
join the AAA and attend the next meeting." 



b. Unenthusiastic : "Only in how to increase the effectiveness 

of such a workshop in the future." 



3 . In terms of the entire session, in both its formal and infoimal 
aspects, which activities (and/or individuals) were most help- 
ful for your purposes? 

a* Enthusiastic : "Different people were helpful in different 

ways : 

Spindlers — the theories and designs of anthropological re- 
search 

Wolcott— ethics, field work problems 

L. Smith — special help directly applicable to my research 
A. Smith— technology and personal attitudes toward anthro- 
pology 

Participants— conceptualization of general and specific 
ideas." 



b. My favorite : "Chairman was the best I’ve ever seen." 

k. Which activities (and/or individuals) were least helpful for 
your purposes? 

a. Enthusiastic: "After having attended the session last 

year, I feel I have to react to the total program this 
year. The activities and personalities all supported each 
other. I know it was partly accidental, but the program 
came off, for me, in a way that was very beneficial— in 
total." 



b. Unenthusiastic: "1. 

2 . 

3 . 

4 . 

5 . 



Much overlap in activities. 

Wot enough coherence or order. 

Too much ’presentation of self’ of 
panel. 

Too rigid in their self-interests. 
There were few informal aspects— un- 
less it was the coffee break. 
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6. We might have had materials by 

other — not present — anthropologists 
(or sociologists) to acquaint the 
group of educators. Why this narrow 
range of perceptions and operations?" 



Directors Evaluation. My feeling is that the session was a splen- 
did one. We seemed to have our share of dippy people, malcontents, and 
dropouts, but we also had many very bright and deeply interested partici- 
pants. In general, I think the morale of staff and participants remained 
very high through, the session. In spite of a lack of consensus in the 
conments of participants, I think the success of the session was due in 
part to the fact that everyone, staff and participants alike, knew exactly 
where we were going in terms of the content and organization of the ses- 
sions. As director I confess to conducting the sessions with a firm and 
heavy hand regarding the operation of the meetings. My purpose was to 
maximize the time available for substantive presentation and discussion. 

The staff was superb. They were eminently qualified in presenting 
the content of the sessions. Prior to the sessions we had identified 
what contribution each person felt he could best make, and each member 
had wade careful preparation for those sessions in which his role was the 
instructional one. In addition, all staff were present at all sessions 
to participate in a continuing dialogue regarding the discussions. Staff 
made themselves freely available for meeting individually with partici- 
pants, and participants expressed their appreciation for the whole-hearted 
extent of staff participation and commitment. 

Several "working rules" contributed to the organizational success 
of this session: 

1. Instructional sessions were relatively formal but never lasted 
over an hour without a change of pace (e.g., discussion, panel 
reaction) or a break. Formal meeting time was limited to two 
3-hour sessions a day, promptly begun and promptly terminated. 

2. Arrangements were made to have coffee catered at each session. 
The arrival of coffee signaled an "instant" break. 

3. Thanks particularly to David Boynton of Holt, Rinehart and 
Winston, as well as to individual staff members. Copies of 
case studies and other recent printed and mime ographed materi- 
als were available and were widely circulated among partici- 
pants throughout the sessions. 

4. Considerable variation was offered in the formal program and in 
the personalities and commitments of individual staff memb ers. 

The extent of correspondence between participants and staff reported 
since the session suggests that participants may have found more help and 
common ground with individual staff members than they themselves realized 
on the last morning of the sessions. One wonders, in reading over their 
criticisms and suggestions for next year, if there is a distinction 



between a “training aeasion" and a ’ Wahop" 

fg % ^rse^oTe^sSS^garaing a get-acquai^ed party a^ 
desired “problem-clinic" approach to ^"kLS in^ 

more topics and seme ,,^ r ; e f s fJ^ r ^fto SISde any 

vhile others touted ix the best session ever ^ 

director fro© trying to satisfy everyone, 
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PRE SESSION VIII: SAMPLE FREE TEST 

AKD PERSON MEASUREMENT IN EDUCATIONAL RESEARCH 



Staff 



Benjamin Wright 
(Director) 

Clarence Bradford 



University of Chicago 
Chicago, Illinois 

University of California 
Los Angeles, California 



General Description 

The topic treated in this presession was mathematical “dels for 
objective measurement. Applications in educational research vere 
stressed. A new approach to psychological measurement which «<*dev 
•rfnLortant kind of measurement objectivity was explained and Ulus- 
trated Illustrations focused on the dichotomous observations 10 
rharacterize^rdinary item analysis, hut generalizations to more expli- 
cated observations such as frequencies and polychotonn.es were ^ c ^ se 
Participants acquired an overview of this approach 

eluding a recognition of the importance of measurement ob '^ctivity^and 
an elementary mastery of how logistic models cjme ^ et>od to 

surement problems. They worked through applications of th — 

■rpftl data and applied the method to their own data if they wished. 
I^icip^rtlsf had a chance to work on smasurement problems during 

the session. 



Objectives 

Because of the diversity in interests and mathematical training 
only f few broLly defined objectives could be specified for the whole 
mono. To expect all participants to follow the mathematical an 
retical aspects of the model was unreasonable. Yet, these aspects of 
Se ^Sel SSd not he omitted. To resolve this the staff presented ma- 
terial^enco^assing the entire spectrum of relevant topics-measurement 
theory, mathematical derivations, and applications to data. In this 
the Sin objective was to convey an overview of this new approach to 
Sasurement^ including a recognition of the importance 
oblectivity and an elementary mastery of how logistic models 

colve measurement problems. The second objective was for partici- 
pants to gain an understanding of the potential usefulness and appliea- 

tions of the method to real data. 



Schedule 



Day I 

Session 1: A Theory of Measurement 

Session 2: Probability Structure of the Model 
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Session 3 : 


Measurement Tutorial 


Day II 




Session 1: 


Logistic Estimation 


Session 2: 


Mean Value Estimation 


Session 3 • 


An Example of Item Analysis 


Session 4: 


Estimation Tutorial 


Day III 




Session 1: 


Control of the Model 


Session 2: 


Item Pools and Test Designs 


Session 3 ‘ 


Necessity of the Model 


Session 4: 


Control Tutorial 


Day IV 




Session 1: 


Problems and Possibilities 


Session 2: 


Objective Measurement in Developmental Studies 


Session 3» 


Programming the Model 


Day V 




Session 1: 


Measurement and Objectivity 



Participants 

The 46 participants attending the presession represented a wide 
range of abilities and interests. Thirty were primarily interested in 
the application of the measurement model to real data. The remaining 
16 were not only interested in applications but also in working through 
the mathematical theory and derivations of the model. 

Fourteen states, Puerto Rico and Canada were represented. Over 25 
percent of the participants came from the West Coast, and 13 percent 
from Canada. Nineteen universities and six business and government in- 
stitutions were represented by participants. Seventeen percent of the 
participants were from business organizations dealing with testing. 
Twenty-six percent were associated with research associations and eval- 
uation centers. Three government institutions were represented by par- 
ticipants. 



Materials 



Examples of the materials used were: Schmidt, A Theory of Measure- 

ment ; Keesling, Evaluation of Fit, Control of the Model ; Wright, Sample- 
Free Test Calibration and Person Measurement ; Wright, Mean Value Esti- 
mation; and others. 
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Evaluation 



Test Results . Evaluation in such a heterogeneous situation was dif- 
ficult. To handle the diversity of the group, a dual scheme was adopted. 
First, general objectives were established by the instructional staff, 
and three questions were asked to assess the attainment of these objec- 
tives. Second, participants were asked to define their own objectives 
and reasons for attendance and then to assess how well they felt their 
own goals had been realized during the presession. 

Thirty-two participants responded to the three questions asked. 

(The remaining 14 participants were absent from the meeting in which 
these questions were asked. Most of these l4 were either working on 
their own data or involved in a session on computing problems.) The 
first question, '*What new and important idea did you gain from the pre- 
session?, " dealt primarily with the first objective. The results sug- 
gested that all 32 of the participants who responded had gained a gen- 
eral overview and elementary grasp of this new approach to measurement. 

The second question, "How will you use the ideas you gained from 
the presession?," provided insight for both objectives. The responses 
fell into three categories: teaching, statistical research, and test 

construction. • Twelve and five tenths percent would use the ideas gained 
in teaching, 37-5 percent in statistical research, 4l percent in test 
construction, and 9 percent found no use. 

The fact that approximately 78 percent of these people intend 
either to do research on the applicability of this model to different 
kinds of data or actually to apply the technique in their professional 
activities as test constructors was very heartening. 

Twenty of the participants brought data from their own research to 
work with. A total of 63 computer runs were made on these data. The 
progressive application of the model to data by participants suggests 
that they were able to apply the technique to real data in a rewarding 
way. 



The third question was, "If you were to give a presession on sample- 
free measurement, what improvements would you make or what would you re- 
tain from the present presession?" Approximately 37 percent volunteered 
comments indicating their satisfaction with the lectures. Sixteen per- 
cent expressed their satisfaction with the opportunity provided in the 
presession for individual assistance. 

On the negative side six participants indicated their dissatisfac- 
tion with a physical feature of the presession. They didn't like the 
motel. The lecture room was too small. Two felt that the turn-around- 
time at the computer was too long. Four felt that more data examples 
should have been given. 

The second evaluation scheme produced similar data. The organiza- 
tion of the presession was such that a wide range of topics was pre- 
sented so that people could concentrate on any one of these areas or all 
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of them. Thus, one evaluation of the success of the presession would he 
the degree to which the participants themselves felt they had realized 
their own objectives. The participants were asked in the beginning of 
the presession to list their own objective for attending the presession. 
This was done before the staff presented their proposed objectives. At 
the end of the week’s activities each person was asked to evaluate how 
well his objectives had been attained. The data are summarized below 
in Table 12. 



Table 12 

Participants* Objectives 



Objective 


No. of people 
listing the objective 


No. of people 
feeling they attained 
their objective 


To get an overview of this 
new technique for item 
analysis 


23 


23 


Tc become acquainted with 
applying the model to item 
analysis data 


20 


19 


To study the theory and 
mathematics of the model 


15 


15 


To study the extension of 
the model to polychotomous 
data 


5 


0* 


To learn a new theory of 
measurement 


k 


3 


To study test items for 
which the model is appli- 
cable 


3 


3 


To look at the implications 
of the model for item pools 


3 


3 


To learn the relationship of 

this model to other models 

for item analysis 2 


2 


To evaluate the robustness 
of the model 


2 


2 



*This topic was not presented. 
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The oblectives listed by the participants provide a range 
jectives forsuch a presession. The frequency distribution su^rts 
the flexible organization that was employed. These data suggest th 
the leSing activities provided by the staff were 

satisfy the participants to a large degree even though the P^ie ipgts 
t hemse lves represented a heterogeneous group in both interest and abil 

ities. 

Director's Evaluation. The presession was hard work. The variety 
of oarticinants required preparation of materials on several levels. 

To accoonlish this we had a Series of preliminary working sessions in 
Chicago to prepare the instructional staff to deal with both n ®J- Te 
SSic,W participants. We also prepared detailed instruction^ 
materials to make it easier for participants who wished to becOK Jtor 
Sophisticated to follow the presentation. Finally.wearranged to have 
simult aneous sessions at different levels of sophistication. The re 
suits of these plans were quite satisfying. Participants wereat many 
levels of interest and sophistication. Nevertheless we were able to 
provide something useful for each of them. 

Our chief problem was obtaining adequate computer services. In 
suite of extensive preparations in Chicago and Los Angeles, we still 
were delayed in providing computer output for those twenty participants 
who brought their own data to analyze. If there was any short caning 
in the presession this was it. Although we did 63 compeer runs, 
is an average of more than three for each participant who brought 
data, still I do not feel satisfied with this aspect. Should we repeat 
this presession I would invest further efforts toward assuring rapid 
computer turn-around so that each participant could have four or five 
rounds of data analysis according to this item analysis model. 

One outcome of the presession which was not anticipated and which 
may testify to its impact was the spontaneous creation of an informa- 
tion center and clearinghouse for this kind of item analysis. Dr. 
Alexander Even, of the Ontario Institute for Studies in Education, vol- 
^S tfbe’the coordinator and organized a group of participants who 
wished to communicate through him with each other on problems of mea- 
surement and item analysis. 

Participants* A nonymous Evaluations . On the 
sion 26 participants completed an anonymous evaluation form which 
administered by’and returned directly to the 1969 AERA Hesearch Train- 
ing Sessions Committee. Four items on that fom pertained directly to 
the quality of the session. Responses to these questions follow. UJ 
Please rate the quality of instruction in your session. Excellent 
Good 12$, Average 0$, Fair 0$, Poor 0$. (2) Leaving asi e * ^ ^ 

of instruction for the moment, do you think the ^2. “ 1 ain 

session should be treated again next year? Yes 85$, No Cfy, Unc ^ a ~\ 
15$. (3) If you had it to do over again, would you apply for the ses- 
sion which you have just completed? Yes 100$, No ^ 

(k) If a session such as this is held again, would you recanmend to 
others like you that they attend? Yes 100$, No 0$, Uncertain 0$. 
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POSTSESSION Is SURVEY 
RESEARCH IN EDUCATION 



Staff 



James G. Anderson 
(Director) 

Harlley E. McKean 



New Mexico State University 
Las Cruces, New Mexico 



New Mexico State University 
Las Cruces, New Mexico 



Harry R. Potter 



Purdue University 
Lafayette, Indiana 



General Description 

The postsession was a special training program in survey research 
techniques aimed at demonstrating the use of the survey as a primary 
instrument for scientific study in education. 

Survey techniques have not "been generally applied to the study of 
education, albeit they have been utilized most effectively by social 
scientists for more than 25 years. This limited use of survey techni- 
ques derives from the circumstance that such techniques have been pre- 
sented almost entirely as means of collecting descriptive information 
and little attention has been devoted to the analytical aspects of sur- 
vey research. Both Trow and Sieber vividly underscore this assertion 
by quoting texts on educational research which, at best, devote limited 
space to discussions of the clerical aspects of surveys and pointedly 
exclude any discussion of the analysis of survey data; at worst, in- 
veigh against nflin g survey methods for causal inferences concerning be- 
havioral phenomena. 

This distrust of survey methods is manifest despite the fact that 
by far the major proportion of empirical generalizations in sociology 
during the past two decades have been derived from survey studies. One 
has only to look at Lipset, Trow, and Coleman* s study of the adolescent 
subculture of the high school; and Gross, Ward and McEachern*s study 
of the role of the school superintendent, to appreciate the sophisti- 
cated use of survey techniques made possible by developments in sampl- 
ing theory, interviewing techniques, attitude measurement, multivariate 
analysis and computer technology. 

The program was designed to prepare those who had little or no 
previous experience with survey research to apply survey methods to 
problems in education. During the session students were acquainted 
with recent developments in sampling methods, interviewing techniques, 
attitude measurement, and multivariate analysis. The application of 
these research methodologies was illustrated with actual poverty sur- 
veys, cross-cultural surveys, and institutional surveys conducted by 



members of the instructional staff 



During the session students were acquainted with recent develop- 
ments in sampling methods, interviewing techniques, attitude measure- 
ment, and multivariate analysis. The application of these research 
methodologies was illustrated with actual poverty surveys, cross- 
cultural surveys, and institutional surveys conducted by members of 
the instructional staff. 

An actual survey of the participants was conducted to permit par- 
ticipants to gain experience in the design of a survey, the collection 
of data, the creation of scales and indices, and the analysis of survey 
data. 

Objectives 

The program was designed to prepare those who had little or no 
previous experience with survey research to apply survey methods to 
problems in education. Participants were to gain experience in the fol- 
lowing survey research techniques: (l) the statement of objectives, 

(2) the construction of simple scales and indices, (3) questionnaire 
construction, (4) interviewing techniques, (5) data collection, (6) cod- 
ing, and (7) selected methods of analysis. 

Schedule 



Dajr I 

Session 1: 



Session 2: 
Session 3: 

Day II 

Session 1: 
Session 2: 
Session 3: 

Day III 

Session 1: 
Session 2: 
Session 3: 

Day IV 

Session 1: 
Session 2: 
Session 3: 



General Introduction, Introduction to Survey Re- 
search, Class Questionnaire and Coding of Class 
Questionnaire 
Planning Surveys 
Review of Statistics 



Sampling 

Sampling 

Measurement, Guttman Scaling 



Measurement, General; Data Collection 
Data Analysis I, Data Analysis II 
Free Time 



Data Analysis II, Data Analysis III 
Data Analysis III, Data Analysis IV 
Data Analysis IV 
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Day V 

Session Is Graphical Displays, Evaluation of Training Session 



Participants 

Out of the 103 applicants 63 actually participated in the postses- 
sion. The geographical distribution showed 25 percent of the partici- 
pants from the Northeast, 32 percent from the Middle Atlantic, 17 per- 
cent frcsn the Southeast, 20 percent from the Midwest, and 2 percent each 
fraa the Southwest, Canada, and Germany. 

Information on the participants educational background showed that 
67 percent held a doctoral degree. Approximately 2b percent indicated 
that they had attended a previous AEEA pre session. 

si yhy-two percent were affiliated with a college or university, 13 
percent with public school systems, six percent with state departments 
of education, and two percent with the U.S. Office of Education. Seven- 
teen percent were affiliated with other institutions such as religious 
organizations and regional laboratories. 



Materials 

Each participant was provided with a notebook and a set of Case 
Studies. For every set of three persons a copy of Sociological Analysis: 
An Empirical Approach Through Replication by Murray A. Strauss and Joel 
I. Nelson (New York: Harper and Row, 1968) was provided. 

Instruction and evaluation were broken down into ten sections. A 
questionnaire from Murray and Strauss was completed by each participant 
and coded cn the morning of the first day of the training session in 
order to provide data for the class exercises. 



Evaluation 

Test Results . Suggestions and canments were solicited fraa the 
participants on the morning of the last day of the training session, 
categorized summary of these comments is given below. 

Organization 

"It was well-planned and conducted.” 

"Organization O.K.” 

"Organization was very good — handout format excellent. ^ 

"Gear session to either advanced researchers or beginners. 

"Keep whole group instruction to a minimum. Have each instructor 
conduct sessions in his specialty and arrange a round-robin for- 
mat. This would make it possible for sessions to be conducted on 

several levels." 



a 




Content 



"Sampling — good. " 



"More on data analysis by emitting the actual calculation of 
statistics." 

"Place more emphasis on the design of survey research, i.e., in- 
strument construction, coding, interview techniques." 

"More discussion of statistical power of chi square, etc. This 
was done on scaling and should surely have been done on the 
statistics — parametric vs. nonparametric. " 

Computation 

". . .a few calculators would increase efficiency and remove the 
tedious calculations." 

"Computer terminal available for familiarization of what it can 
do." 

"Have desk calculators for tabulation or have this part of opera- 
tion computerized. 

Many class exercises were designed to permit the student to gain 
actual experience in all aspects of designing, conducting and analyz- 
ing the data from social surveys. Furthermore, these exercises per- 
mitted the staff to assess whether or not individual participants had 
mastered a particular research technique (i.e., sampling technique, 
scaling technique, statistical technique, etc.). A list of these ex- 
ercises and the proportion of participants who successfully completed 
each one follows in Table 13* 



Table 13 



Class Exercises 



SUBJECT 



IERCENTAGE 

COMPLETING 

EXERCISE 



PLANNING SURVEYS 



95 



SAMPLING 

Simple Random Sampling 
Cluster Sampling 
Stratified Random Sampling** 
Simple Random Sampling** 



100 

100 

25 

10 



MEASUREMENT 

Social Contact Distance Scale 
Guttman Scaling 
Factor Analysis 




(Continued) 
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Table 13 (Continued) 



SUBJECT 

DATA COLLECTION 

Sociological Analysis Questionnaire* 

Questionnaire Construction 

Interviewing and Interview 
Schedule Construction 

Coding Sociological Analysis 
Que st ionnair e* 

Coding and Tabulation of Survey Data 

DATA ANALYSIS I 

One Variable: 

Use and Computation of Percentages 
Use and Computation of Percentages 

DATA ANALYSIS II 

Relationship Between Two Variables: 

The Use of Percentages 
The Use of Percentages 
The Use of Percentages 

DATA ANALYSIS III 

Relationship Between Two Variables: 
Measures of Association 
Chi Square 

Percentage Difference 
Gama 

Product Moment Correlation 
Partitioning the Degrees of Freedom 
in Contingency Tables** 

Testing for Trends in Contingency 
Tables** , 

DATA ANALYSIS IV 

Relationships Among Several Variables: 
The Use of Percentages 
The Use of Percentages 

Relationships Among Several Variables: 
Measures of Association 
Gamma 

Product Moment Coorelation 

Relationships Amcjg Several Variables: 
The Use of Percentages 
The Analysis of Mean Scores 
Multiple Regression Analysis 

(Continued) 




PERCENTAGE 

COMPLETING 

EXERCISE 



95 

NA 

NA 

95 

NA 



100 

100 



100 

100 

NA 



100 

NA 

100 

100 

^5 

35 



NA 

100 



NA 

NA 



NA 

NA 

NA 

NA 
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Table 13 (Continued) 



PERCENTAGE 

COMPLETING 

S UBJECT EXERCISE 

REVIEW OF ELEMENTARY STATISTICS** 

Descriptive Statistics 50 

Randan Variables 50 

Hypothesis Testing 50 

Comparing Two Means 30 

The Use of Chi Square 3° 

Uses of the F Distribution 30 



(**0ptional Exercise, #NA: Exercise not assigned. ) 



Directors Evaluation . The large number of applicants for this 
training session would suggest that many persons involved in teaching 
and conducting educational research desire an opportunity to learn 
about survey research methodology, and were not exposed to these tech- 
niques during their academic training. 

Two-thirds of the participants held the doctorate. The median 
number of courses in statistics and experimental design completed by 
these individuals was between three and four. It would appear that 
most participants followed a traditional academic program, in which they 
were exposed to several courses in statistics and exper ime nt al design 
with little or no exposure to survey research methodology. 

The fact that forty persons who registered for the postsession did 
not act ually attend can be attributed to several factors. The first 
was the long registration period that preceded the postsession. Twenty- 
three of the 40 notified me of their withdrawal fran the session. Rea- 
sons given varied from illness and injury to campus insurrection. In 
general, unexpected circumstances arose between the time when they reg- 
istered and the end of March that prevented them from attending. 

The second factor in my estimation was the confusion that existed 
regarding the cost to participants and the availability of the stipend. 
Unfortunately, the literature that the University of Maryland sent to 
applicants contradicted the statement made on the official application 
that ’'neither fees nor tuition is charged for any of the sessions. In 
contradiction to this statement the 'registration fee charged by 
the University of Maryland came as quite a surprise and could scarcely 
be defended as covering meals. Many applicants wrote and called me 
questioning this when they received the registration forms from Mary- 
land. Moreover, a number of persons refused to send the fee in advance 
until the availability of the stipend was clarified. 

The letter that went out to applicants fran the AERA central office 
less than a week before the postsessions were scheduled to begin, un- 
fortunately, only further confused the matter. As a result I received 
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several very irate letters of cancellation* I believe that at least 
half of those persons -who registered but failed to attend may have done 
so because of this confusion over the actual costs of attending. In 
the future I strongly recommend that these matters of "registration 
fees” and availability of stipends be clarified before any announce- 
ment is made regarding the research training sessions. 

Aside from the problem concerning fees, the Center of Adult Educa- 
tion at the University of Maryland provided a fine setting for the ses- 
sion. Moreover, the staff of the Center provided excellent support. 
Having such a University Center handle registration and logistics as 
well as the availability of AV equipment, duplication facilities, etc., 
has much to commend it for future research training sessions. 

The session itself went quite well, in my estimation. We accom- 
plished a great deal of what we had intended to sc ccmpli sh. However, 
several of the participants* suggestions that were outlined earlier 
are important for future training sessions based on this topic. For 
example, making available a computer terminal for the session next time 
would serve to introduce participants to the very important technical 
aspects of processing survey data. At the same time use of such a ter- 
minal would cut down on the tedious and time consuming clerical opera- 
tions and calculations that are necessary when analysing data* 

Although, we did attempt to differentiate the program to suit the 
interests and level of participants wherever possible, such differen- 
tiation might easily be incorporated into the program and notebook from 
the very beginning by providing a series of optional and supplementary 
exercises for participants who wish to pursue special topics and/or 
more sophisticated research methodologies such as the use of multi- 
variate statistical techniques in the analysis of survey data and a 
variety of sophisticated scaling techniques such as factor analysis, 
etc. Additional material on a wider variety of data collection tech- 
niques could easily be included in the notebook. In this fashion 
groups of participants could easily simultaneously work on different 
exercises aimed at different levels but all related to the same central 
topic. 

In summary, I believe that the Research Training Session on Survey 
Research proved to be most popular and successful and should be offered 
again next year. In fact, the 40 persons who applied for this year’s 
session but who did not attend for a variety of reasons may well welcome 
an opportunity to participate the next time that the topic is offered 
if their expressions of regret at not being able to attend this year are 
any indication. 

Participants 1 Anonymous Evaluations . On the final day of the ses- 
sion 48 participants completed an anonymous evaluation form which was 
administered by and returned directly to the 1969 AERA Research Train- 
ing Sessions Committee. Four items on that form pertained directly to 
the quality of the session. Responses to these questions follow: (l) 

Please rate the quality of instruction in your session. Excellent 28$, 
Good 51$, Average 19$, Fair 0$, Poor 2$. (2) Leaving aside the quality 



of instruction for the moment, do you think the topic treated in your 
session should be treated again next year? Yes ES % No 6£, Uncertain 
6$. (3) If you had it to do over again, would you apply for the ses- 

sion which you have just completed? Yes 69f>, No 12 $, Uncertain 19 $* 
(4) If a session such as this is held again, would you recommend to 
others like you that they attend? Yes 85/6, No 11 $, Uncertain 4$. 



POSTSESSION II: MULTIPLE GROUP 

DISCRIMINANT STRATEGY 



Staff 



Paul R. Lohnes 
(Director) 



State University of New York 
Buffalo, New York 



William W. Cooley 



University of Pittsburgh 
Pittsburgh, Pennsylvania 



Richard Ferguson 



University of Pittsburgh 
Pittsburgh, Pennsylvania 



General Description 

A natural research paradigm for studies of human development is to 
employ an antecedent personality traits profile as predictor of later 
adjustment outcomes, where the variety of adjustments is described by a 
taxonanic variable. Such studies are surveys, in which the life pat- 
terns of subjects are observed as they unfold naturally. The MANOVA 
statistical procedures for analysis of multivariate experiments are less 
than comprehensive in application to surveys, particularly as MONOVA em- 
phasizes inference rather than description. There is need for substan- 
tial improvement in the statistical strategy for implementing the trait - 
and- factor research paradigm in developmental psychology (D. E. Sup>er, 
et al.. Vocational Development : A Framework for Research, 1957* P» 79) • 

What is needed is a more heuristic strategy than MONOVA; one which in- 
duces the shape and details of relationships in the data. 

For the past fifteen years a series of approximations has been con- 
verging on an adequate solution to the methodological problem of multi- 
variate trait prediction of taxonomic criteria, A group of workers at 
the Harvard Graduate School of Education has been especially influential 
in promoting the techniques of multiple group discriminant analysis and 
classification probabilities (P. J. Rulon, D. V. Tiedeman, M. Tatsuoka, 
C. Langmuir, Multivariate Statistics for Personnel Classification, 1967) 
The former technique has proven to be so central to an adequate solution 
to the problem in the experience of Cooley and Lohnes that they have in- 
corporated its nomenclature in their title for the overall strategy that 
has evolved (W. W. Cooley and P. R. Lohnes, Predicting Development in 
Young Adults, 1968) . 

The elements of the sequence of statistical analyses that Cooley 
and Lohnes now consider to be the most useful strategy for longitudinal 
developmental surveys was the focus of this pre session. Ten years of 
personal experience in researching career development, as well as atten- 
tion to the methodological issues in other research on human develop- 
ment, have contributed to the evolvement of this strategy. The preses- 
sion was intended to explore the "Why? 11 of this strategy thoroughly, as 
well as the "How?” of it* Content was about equally divided between the 



statistical models themselves and examples of actual research applica- 
tions. Students were encouraged to discuss their own research experi- 
ences and plans in relation to this strategy. Instructors treated this 
as a fluid rather than a final strategy and urged creative thinking 
about alternative approaches. 

One feature of the presession was that it introduced students to 
a set of multivariate statistical computer programs in the FORTRAN lan- 
guage which Cooley and Lohnes have recently written as part of the prep 
aration of a second edition of their hook (W. W. Cooley and P. R. 
Lohnes, Multivariate Procedures for the Behavioral Sciences, 1962). 
Lohnes made these programs available to students who sent a tape to 
Buffalo. These programs should be of significant assistance to stu- 
dents who want to apply the strategy in their own research. 






Objectives 

The objectives of the session were the following: 

Objective 1: To promote insight into the epistemological problems 

of the science of human development. 

Objective 2: To promote mastery of a specific strategy of research 

methodology. 

Objective To promote understanding of a specific set of multi- 
variate statistical models. 

Objective kz To promote skill in the use of a specific library 
of computer programs. 

Schedule 



Day I 



Session 1: Epistemological Problems of Research on Human 
Development 

Session 2: Epistemological Problems 

Session 3: Overview of a Research Strategy 



Day II 



Session 1: Organizing the Predictor Data: Factor Analysis 

Models 

Session 2: Research Examples 

Session 3: Research Examples and Alternative Approaches 



Day III 



Session 1: Organizing the Criterion Data: Distance Function 

Models 
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Session 2: Research Examples 

Session 3: Examples and Alternatives 



Day IV 

Session 1: 



Session 2: 
Session 3: 



Day V 



Session 1: 
Session 2: 



Predicting the Criterion: Multiple Group Discrim- 

inant Functions Model 
Research Examples 
Examples and Alternatives 



Evaluating the Obtained Prediction Model: 
fication Statistics 
Research Examples 



Classic 



Session 3: Examples and Alternatives 



Participants 

Distribution of participants revealed 90 percent male and 10 percent 
female and an average age of 35 years. Sixty- five percent of the par- 
ticipants held the doctorate, and 33 percent held a master f s degree. At- 
tendance at previous AERA presessions was indicated by 33 percent of the 
participants. Research productivity of the postsession group was re- 
vealed in the averages of three articles published in scholarly journals 
and .9 funded project per participant. 



Materials 



The materials for the postsession included two books by W. W. Cooley 
and P. R. Lohnes, Multivariate Procedures for the Behavioral Sciences 
(N. Y. : Wiley, 1962), and Predicting Development in Young Adults (Palo 
Alto: American Institutes for Research, 1968 )• There were approximately 

three hours of IBM 360 time-sharing computing of multivariate statistics 
in dialogue mode and approximately three hours of batch-processing mode 
computing on IBM 709*+ • 



Evaluation 



Test Results. Two evaluation procedures were employed. First, the 
director will keep a log of requests for the computer programs generated 
by the presession. The person requesting program listings will have to 
make the effort to send a tape to Buffalo, which will indicate seme real 
interest in the procedures studied. Second, students were strongly urged 
to forward to the director any research reports and reprints incorporat- 
ing seme or all of the strategy taught in the presession. The director 
has committed himself to prepare a report on the concrete evidence for 
such effects over a four-year period. 

Director’s Evaluation . The three instructors agreed that they 
learned a great deal from the students, and that they will be busy for 
several years exploring the excellent suggestions made by students for 
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modifications in and supplements to the research strategy covered by 
the session. The students had studied both texts quite thoroughly an< ^ 
wqyq well-prepared for the session* Most of them turned out to be ac- 
tively involved in research programs in which they are trying to apply 
the discr iminant research strategy. With few exceptions, the students 
appeared to be appropriate for the session, participated actively, 
seemed to be enjoying themselves, and appeared to be learning the re- 
search strategy. 

Over half of the students brought research data decks to the ses- 
sion, and about two-thirds actually computed analyses during the ses- 
sion. This was one of the big surprises to the instructors, who had 
invited students to bring data but didn’t expect many to do so. Every 
evening of the session there were vigorous small-group discussions 
about these actual research computations. The two extensive demonstra- 
tions of real-time computing of discriminant analysis via teletype com- 
munication with the University of Pennsylvania Computing Center seemed 
to be of great interest to the students. The provision of these com- 
puter experiences seemed to be a major positive value in the program. 
Both the University of Maryland and the University of Pittsburgh are to 
be commended for contributing the computer time, and special commenda- 
tion is due to Bichard Ferguson of the instructional staff, who worked 
far into several nights of the session, making the necessary adjust- 
ments to the strange environment of the Maryland Univac. 

The training site was ideal, except for the distance to the Com- 
puting Center. The living arrangements were also ideal, except for 
the lack of planning for same socials. With the small effort needed 
to ease the local transportation problem and to schedule same socials, 
the Center for Adult Education would be ideal for future sessions. The 
Center staff was great! 

There was no AERA presence to speak of. Another time an effort 
should be made to get the President of AERA out to greet the troops. 

The length-pace trade-off for these sessions needs to be examined. 
The 4-1/2 days might work better if several evening socials were ar- 
ranged, but I suspect that 3-1/2 days with evening training sessions 
might make more sense. There is need to insist that people arrive the 
night before the first morning session, or else start at 1:00 p.m. the 
first day. We had a mental fatigue factor set in this time. 

Participants ’ Anonymous Evaluations . On the final day of the ses- 
sion 29 participants completed an anonymous evaluation form which was 
a dmin istered by and returned directly to the 1969 AERA Research Train- 
ing Sessions Committee. Four items on that form pertained directly to 
the quality of the session. Responses to these questions follow: (1) 

Please rate the quality of instruction in your session. Excellent 31%> 
Good 3&f), Average 1 ? % Fair 7 % Poor 7$- (2) Leaving aside the quality 
of instruction for the moment, do you think the topic treated in your 
session should be treated again next year? Yes 90%, No 7%> Uncertain 
3 (3) if you had it to do over again, would you apply for the 



session which you have just completed? Yes 80 $, No 17$, Uncertain 3$. 
(4) If a session such as this is held again, would you reconnend to 
others like you that they attend? Yes 83$, No 10$, Uncertain 7$- 




POSTSESSION III ; BAYESIAN 
STATISTICAL ANALYSIS 



Staff 



Donald L. Meyer 
(Director) 



Syracuse University 
Syracuse, New York 



-Jaimes E f Powers 



State University of New York 
Albany, New York 



General Description 



The general topic was the use of Bayesian methodology in designing 
and analyzing statistical-type studies. Specific topics included the 
philosophy and rationale of Bayesian statistics including an introduc- 
tion to subjective probability functions for hypothesis testing and 
estimation, the problem of determining optimal sample size, specific 
applications to the general linear model and associated analysis of 
variance techniques. 



Objectives 



At the close of the session the participants were to be able to; 

Objective 1: Read and evaluate the current literature on Bayesian 

methodology. 



Objective 2; Apply general Bayesian principals in their studies. 

Objective 2.: Appreciate Bayesian philosophy and logic with special 
reference to inference. 



Objective k: Gain insights into classical statistics as a result 

of exposure to a competing philosophy and will be 
able to better teach statistics to others. 



Schedule 

Session 1; Introduction to Bayesian statistics. Philosophy, 
definitions of probability 

Session 2; Basic probability axioms and theorems, Bayes* 
theorem, problem session 

Day II 

Session 1; Comparison of classical and Bayesian approaches— 
relation of errors and losses 
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Session 2: 

Day III 

Session Is 

Session 2: 



One state and many states of nature, likelihood 
ratios, problem session 



Bayesian inference using Beta distribution, high 
density posterior probability intervals, natural 
conjugate Bayes densities 

Optimal, sample size and loss functions, minimax 
rules, inadmissable rules, lab session 



Day IV 



Session 1: Problem session, normal sampling processes, 

inferences on means 

Session 2: Analysis of linear models, inferences on both 
means and variances 



Day V 



Session 1: 



Miscellaneous questions, attitude survey, error 
rates in sample surveys 



Participants 

Information from 31 participants revealed that 75 percent were male 
and 25 percent female. The average age for the group was 35 years. The 
majority of the participants (71 percent) were employed at a college or 
university. Three percent were from the public school systems, seven 
percent from the federal government, seven percent from state depart- 
ments of education and 12 percent from other sources. Twenty- five per- 
cent had previously attended an AERA presession. 

Academically, 61 percent possessed the doctorate and 32 percent 
possessed a master* s degree. Seven percent held neither. As a group, 
they had an average of three articles published in scholarly journals 
and .7 funded project. 



Materials 



The materials used were xerox or ditto copies of Bracken* s tables 
of beta function, tables of inverted gamma function, tables of binomial, 
and error rates in sampling designs. The students were also provided 
worksheets on optimal sample size. 

Evaluation 



Test Results . An attitude survey was administered by the postses- 
sion staff on the last meeting of the session. The items were checked 
by the participants anonymously. A summary of the results are tabulated 
below: • ~ 



Attitude Survey 



o 

ERIC 



1. Are you more sympathetic to the "classical" approach or to the 
Bayesian approach to statistics? (Mark an X on the continuum 
below. ) 



2« Do you anticipate doing more study of Bayesian statistics? 

Definitely yes 21, probably 6, probably not 1, definitely no 

0 . 

3* If you teach statistics or research methods, will you introduce 
some Bayesian concepts to your students? 

Definitely yes 11, probably 8, probably not 0, definitely no 
0. I do not teach statistics 9» 

4. Do you think that the addition of Bayesian statistics to your 
current statistical knowledge will aid you in your research 
work? 

Definitely yes 17, probably 9, probably not 0, definitely no 1. 
Omit 1. 

5* Do you think you will apply Bayesian statistics or concepts in 
your research work? 

Definitely yes 1 6, probably 9, probably not 0, definitely no 1. 
Omit 1. 

6. Do you think you have a better understanding of "classical" 
statistics as a result of the postsession? 

Definitely yes 18, probably 7, probably not 2, definitely no 1. 

Two free-response questions asking for general reactions elicited 
the following comments: 

a. good presentation 25 

b. well-organized 13 

c. liked philosophical discussion 
contrasting Bayesian and 




Classical 



Bayesian 



classical approaches 

d. good coverage of material 

e. whetted appetite for more study 

f. more hand-out needed 

g. need more applications 

h. more problem sessions 

i. too much material in short time 
notation hard to follow 



5 

1 

2 

10 

7 

4 

5 
3 
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k. mathematical derivation 



difficult 

l. more class discussion 

m, too little philosophy 



2 

1 

1 



Director’s Eval uation * We were generally pleased with our post- 
session. The participants were eager to learn and enthusiastic about 
the subject matter. The facilities were satisfactory and conducive to 
learning. 

The problem of heterogeneity of the group relative to background 
was evident, but did not impede progress significantly. However, it 
resulted in a less than optimal experience for all of the students. 

We have not yet resolved this problem. 

Frcm conversations with the participants it seems that the five- 
day session is about right since less time would result in too fast a 
pace and more time would result in over-saturation. An informal so- 
cial gathering of all participants the second night would have bene- 
fited both the participants and AERA. 

Participants’ Anon ymous Evaluations . On the final day of the ses- 
sion 27 participants completed an anonymous evaluation form which was 
administered by and returned directly to the 1969 AERA Research Train- 
ing Sessions Committee. Four items on that form pertained directly to 
the quality of the session. Responses to these questions follow: 

(l) Please rate the quality of instruction in your session. Excellent 
56%, Good 40%, Average 0%, Fair 4%, Poor 0%. (2) Leaving aside the 

quality of instruction for the moment, do you think the topic treated 
in your session should be treated again next year? Yes 100%, No 0%, 
Uncertain 0%. (3) If you had it to do over again, would you apply 
for the session which you have just completed? Yes 81%, No 4%, Un- 
certain 15%. (4) If a session such as this is held again, would you 

recommend to others like you that they attend? Yes 89%> No 4%, Uncer- 
tain 7 %• 
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POSTSESSION IV: DESIGN AND 

ANALYSIS OF COMPARATIVE EXPERIMENTS 



Staff 



Kenneth Hopkins 
(Co-Director) 



University of Colorado 
Boulder, Colorado 



Jason Mi.llman 
(Co-Director) 



Cornell University 
Ithaca, New York 



Donald T. Campbell 



Northwestern University 
Evanston, Illinois 



General Description 

Topics included in the session were sources of internal and external 
validity; the number, nature, role and complexity of variables used in 
experiments. Also included were the relation of statistical analyses to 
experimental designs; rules of thumb for the analysis of complex, but 
balanced experimental designs; the analysis of unbalanced designs or de- 
signs in which statistical assumptions are not met; quasi-experiment al 
designs; the analysis of covariance; and the techniques of multiple and 
planned comparisons. 



Objectives 

The general objective of the postsession was to increase the com- 
petence of educational researchers in matters of the design and analysis 
of comparative experiments. This objective was reflected behaviorally 
by the mastery tests, problem sets, and inventory which formed a part of 
the instructional materials. 



Some of the cognitive objectives included promoting the partici- 
pants* ability to: 

Objective 1: Explain how randomization and blocking function to 

minimize bias and how blocking further results in in- 
creased precision. 

Objective 2: Distinguish between the various purposes for and types 

of replication. 



Objective j}: Graph and interpret various degrees of interaction. 

Objective 4 : Describe sources of internal and external experimental 

invalidity and designs which minimize their effect. 



Objective : 



Interpret the threats to internal and external valid- 
ity related to a given experiment. 
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Objective 6: State the assumptions underlying a given analysis 

and describe the effects of failure to meet a given 
assumption (e.g., heterogeneous variances, non- 
noimality, independence of measurements). 



Objective 7i Identify the experimental unit involved in any ex- 
periment and compare to the unit of statistical 
analysis. 



Objective 8: Identify appropriate analytic techniques for de- 

signs with disproportional cell frequencies. 

Objective £: Distinguish between main and nested classifica- 

tions and between fixed, random, finite, and mixed 
models. 



Objective 10: Interpret an ANOVA summary table and identify the 

design used. 

Objective 11 : Distinguish between planned and ex-post-facto com- 

parisons and between experiment-wise and compari- 
son-wise error rates; compute orthogonal contrasts, 
Nevman-Keuls, Scheffe, and Tukey tests. 



Schedule 



Day I 



Session 1: 



Session 2 



Session 3; 



Day II 



Session 1: 
Session 2: 
Session 3; 



Day III 

Session 1: 
Session 2; 



Types and functions of variables 
Methods of variable control 

Sources of internal invalidity, with emphasis on 

the regression effect 

Internal invalidity 

Statistical interactions 

Sources of external invalidity 



Causal inferences from nonexperiment al data, 
noneorrelational techniques 
Causal inferences from nonexperiment al data, 
correlational techniques 

General discussion, "experiments" in social set* 
tings 



Introduction to classical analysis of variance 
model 

Rules of thumb for the analysis of complex, but 
balanced, experimental designs 



Day IV 



Se»ssion 1: 
Session 2: 
Session 3: 



Analysis of unbalanced designs or designs in which 

ANOVA assumptions are not met 

Violations of assumptions 

Multiple and planned comparisons 

Elective activity 



Day V 

Session 1: Analysis of convariance 

Participant attitude inventory 



Participants 

Ninety-one percent of these participants were male and nine percent 
female. The average age was 37 years. Twenty-three percent o e 
group had previously attended AERA presessions. 

Sixty-six percent were from a college or university and 13 percent 
frcm public school systems. Eight percent were from federal government 
positions, nine percent from state department of education, and four 
percent from other positions. 

The percent of participants holding the doctorate was 71 while 25 
percent held a master's degree and four percent held neither. The re- 
search productivity of the group was reflected m an average of two ar- 
ticles published in scholarly journals ana .9 funded project per par- 

ticipant. 



Materials 

Each participant was asked to bring Experimental and xpepi^ 

mental Designs for Research by Donald T. Campbell and Julian C. Stanley 
"(^ndTMcNally, 19&J". Upon arrival at the post session site, ^ach par- 
ticipant received a loose-leaf notebook which contained all the materi- 
als (demonstration and expository papers, problem sets, evaluation in- 
struments, scrap paper) he needed. Included were severri^bij^ed 
materials such as "The Design of Experiments by 

Linear Model: Lecture Notes" by Kenneth Hopkins, and Problem Set on 

External Validity" by Glenn H. Bracht. Examples of some of Jie published 
materiSs used were: "Rules of Thumb for Writing the ANOVA Table" by 

Jason Millman and Gene V Glass, "Regression and nhe etching FaUacy in 
Quasi-Experimental Research" by Kenneth Hopkins and Reforms as Expert- 
ment" by Donald T. Campbell. 



Evaluation 

Test Results. At regular intervals during the postsession, short 
quizzeFwere administered. The participants recorded their answers on 
a separate cover sheet which they handed in. They were then able to keep 
the test proper while the questions and answers were discussed by the 
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instructional staff* These tests were not mastery examinations* 

Rather, they were composed of more difficult items which served as 
a learning experience. 

Directors Evaluation . In addition to the observations of the 
directors, the two graduate assistants were individually interviewed 
for a dd i tional reactions. The following observations represent some- 
thing of a consensus. 

The postsession appears to have achieved its major objectives 
quite well, namely, to create an awareness of the more critical and 
recently developed concepts and techniques of experimental design and 
analysis as they are related to educational research. There were, how- 
ever, aspects that did not develop as anticipated. There was wide- 
spread dissatisfaction with the contribution of one of the guests in- 
structors who was directed to present techniques for "teasing out" cau- 
sation frcm correlational data. Instead of giving the "main stream" 
approach as we had intended, he presented (in inordinate detail) his 
own work, a very narrow aspect of the general approach. He ignored 
the rather extensive guidelines given to him as well as suggested in- 
structional materials. 

There appeared to be a general feeling that participants need more 
time for follow-up study and completion of the problem sets which fol- 
lowed each major topic. It was suggested that by reducing the time 
given to guest instructors, more time could be allowed for this pur- 
pose. 



The instructional facilities were very good— better than for the 
two previous presessions. The eating arrangements were only marginally 
adequate. Food quality and variety (no menu) was lacking. Many par- 
ticipants felt good quality was not commensurate with cost. The general 
assistance personnel at the Center was excellent. 

The uncertainty of the $75 living allowance was most unfortunate. 

It is felt this reduced the number of participants by 15 to 20 percent. 
This factor also caused much extra work for the directors— answering 
letters and phone calls. The removal of the detail work regarding ar- 
rangements, etc., was a major improvement over previous years. 

The semi-isolation frcm big city night life was a desirable fea- 
ture of the site, although more facilities for recreational activity 
(ping pong, pool, etc. ) in the evening would have provided some needed 
opportunity for a change-of-pace. The lack of facilities for after- 
dinner snacks, etc., was a common complaint . 



Participants * Anonymous Evaluations . On the final day of the ses- 
sion 50 participants completed an anonymous evaluation foim which was 
administered by and returned directly to the 1969 AERA Research Train- 
ing Sessions Committee. Four items on that form pertained directly to 
the quality of the session. Responses to these questions follow: (l) 

Please rate the quality of instruction in your session. Excellent 64 $, 
Good 32$, Average 2$, Fair 0$, Poor 2$. (2) Leaving aside the quality 
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of instruction for the moment, do you think the topic treated in your 
session should he treated again next year? Yes 99&> No 2 Uncertain 
0$. ( 3 ) If you had it to do over again, would you apply for the ses- 
sion which you have just completed? Yes 92$« No h% Uncertain k%. (4) 

If a session such as this is held again, would you recommend to others 
like you that they attend? Yes Skfo, No 2%, Uncertain b%. 



